[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:05:20 dist-test-slave-1gc3 krb5kdc[3938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:05:20 dist-test-slave-1gc3 krb5kdc[3938](info): set up 2 sockets
Sep 24 16:05:20 dist-test-slave-1gc3 krb5kdc[3938](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 16:05:21.693696 3931 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:05:21 dist-test-slave-1gc3 krb5kdc[3938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:05:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:05:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:05:21.902890 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:40951
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:40535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:40951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:22.257865 3952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:22.258261 3952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:22.258368 3952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:22.266044 3952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:05:22.266238 3952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:22.266296 3952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:22.266362 3952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:05:22.266440 3952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:05:22.276971 3952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:40535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:40951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:40951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:22.278465 3952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:22.279836 3952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:22.293887 3958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:22.294064 3957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:22.293913 3960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:22.295015 3952 server_base.cc:1034] running on GCE node
I20260924 16:05:22.295989 3952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:22.297669 3952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:22.298950 3952 hybrid_clock.cc:648] HybridClock initialized: now 1790265922298881 us; error 52 us; skew 500 ppm
I20260924 16:05:22.299407 3952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:22 dist-test-slave-1gc3 krb5kdc[3938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265922, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:22.306988 3952 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:05:22.309044 3952 webserver.cc:460] Webserver started at http://127.3.214.254:38087/ using document root <none> and password file <none>
I20260924 16:05:22.309912 3952 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:22.310055 3952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:22.310426 3952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:22.314060 3952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "62e0a334826e4dd495695c0c4a11ec63"
format_stamp: "Formatted at 2026-09-24 16:05:22 on dist-test-slave-1gc3"
server_key: "bd80c3b3bb688d3424dfeaf633bd1ca1"
server_key_iv: "4b3b3b118f533a2d8591d86527e0a969"
server_key_version: "encryptionkey@0"
I20260924 16:05:22.315049 3952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "62e0a334826e4dd495695c0c4a11ec63"
format_stamp: "Formatted at 2026-09-24 16:05:22 on dist-test-slave-1gc3"
server_key: "bd80c3b3bb688d3424dfeaf633bd1ca1"
server_key_iv: "4b3b3b118f533a2d8591d86527e0a969"
server_key_version: "encryptionkey@0"
I20260924 16:05:22.321943 3952 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 16:05:22.326542 3967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:22.328415 3952 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:05:22.328687 3952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "62e0a334826e4dd495695c0c4a11ec63"
format_stamp: "Formatted at 2026-09-24 16:05:22 on dist-test-slave-1gc3"
server_key: "bd80c3b3bb688d3424dfeaf633bd1ca1"
server_key_iv: "4b3b3b118f533a2d8591d86527e0a969"
server_key_version: "encryptionkey@0"
I20260924 16:05:22.328925 3952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:22.361266 3952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:22.365970 3952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:22.366291 3952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:22.388243 3952 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:40951
I20260924 16:05:22.388247 4018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:40951 every 8 connection(s)
I20260924 16:05:22.389835 3952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:05:22.394832 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 3952
I20260924 16:05:22.395277 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:05:22.395977 4019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:22.398449 3931 external_mini_cluster.cc:1428] Setting key 97aae9999142a71e0ef5c0dc1997368b
I20260924 16:05:22.411772 4019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62e0a334826e4dd495695c0c4a11ec63: Bootstrap starting.
I20260924 16:05:22.415879 4019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62e0a334826e4dd495695c0c4a11ec63: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:22.417289 4019 log.cc:824] T 00000000000000000000000000000000 P 62e0a334826e4dd495695c0c4a11ec63: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:22.420959 4019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62e0a334826e4dd495695c0c4a11ec63: No bootstrap required, opened a new log
Sep 24 16:05:22 dist-test-slave-1gc3 krb5kdc[3938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:22.429841 4019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62e0a334826e4dd495695c0c4a11ec63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e0a334826e4dd495695c0c4a11ec63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40951 } }
I20260924 16:05:22.430433 4019 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62e0a334826e4dd495695c0c4a11ec63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:22.430534 4019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62e0a334826e4dd495695c0c4a11ec63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62e0a334826e4dd495695c0c4a11ec63, State: Initialized, Role: FOLLOWER
I20260924 16:05:22.431331 4019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62e0a334826e4dd495695c0c4a11ec63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e0a334826e4dd495695c0c4a11ec63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40951 } }
I20260924 16:05:22.431651 4019 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62e0a334826e4dd495695c0c4a11ec63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:05:22.431825 4019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62e0a334826e4dd495695c0c4a11ec63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:05:22.432039 4019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62e0a334826e4dd495695c0c4a11ec63 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:22.435453 4019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62e0a334826e4dd495695c0c4a11ec63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e0a334826e4dd495695c0c4a11ec63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40951 } }
I20260924 16:05:22.436127 4019 leader_election.cc:302] T 00000000000000000000000000000000 P 62e0a334826e4dd495695c0c4a11ec63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62e0a334826e4dd495695c0c4a11ec63; no voters:
I20260924 16:05:22.436924 4019 leader_election.cc:288] T 00000000000000000000000000000000 P 62e0a334826e4dd495695c0c4a11ec63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:05:22.437079 4024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62e0a334826e4dd495695c0c4a11ec63 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:22.438967 4024 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62e0a334826e4dd495695c0c4a11ec63 [term 1 LEADER]: Becoming Leader. State: Replica: 62e0a334826e4dd495695c0c4a11ec63, State: Running, Role: LEADER
I20260924 16:05:22.439695 4024 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62e0a334826e4dd495695c0c4a11ec63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e0a334826e4dd495695c0c4a11ec63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40951 } }
I20260924 16:05:22.440116 4019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62e0a334826e4dd495695c0c4a11ec63 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:05:22.441999 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:22.404669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36682 (local address 127.3.214.254:40951)
0924 16:05:22.407152 (+ 2483us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:22.407170 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:22.407236 (+ 66us) server_negotiation.cc:407] Connection header received
0924 16:05:22.416579 (+ 9343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:22.416611 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:22.417133 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:22.417548 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:22.420754 (+ 3206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:22.422289 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:22.424161 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:22.424945 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:22.425174 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:22.431460 (+ 6286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:22.431471 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:22.431490 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:22.431549 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:22.435379 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:22.436462 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:22.436471 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:22.436477 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:22.436714 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:22.437302 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:22.437307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:22.437309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:22.438312 (+ 1003us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:22.438662 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:22.439963 (+ 1301us) server_negotiation.cc:299] Negotiation successful
0924 16:05:22.440472 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2148,"thread_start_us":1824,"threads_started":1}
I20260924 16:05:22.446352 4026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62e0a334826e4dd495695c0c4a11ec63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62e0a334826e4dd495695c0c4a11ec63. Latest consensus state: current_term: 1 leader_uuid: "62e0a334826e4dd495695c0c4a11ec63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e0a334826e4dd495695c0c4a11ec63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40951 } } }
I20260924 16:05:22.446933 4026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62e0a334826e4dd495695c0c4a11ec63 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:22.447467 4025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62e0a334826e4dd495695c0c4a11ec63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62e0a334826e4dd495695c0c4a11ec63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e0a334826e4dd495695c0c4a11ec63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40951 } } }
I20260924 16:05:22.447890 4025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62e0a334826e4dd495695c0c4a11ec63 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:22.450519 4030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:05:22.456274 4030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:05:22.473306 4030 catalog_manager.cc:1347] Generated new cluster ID: e4a9ae3bce334f08b26206cbbfd059f1
I20260924 16:05:22.473465 4030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:05:22.498060 4030 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:05:22.499876 4030 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:05:22.511587 4030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62e0a334826e4dd495695c0c4a11ec63: Generated new TSK 0
I20260924 16:05:22.513002 4030 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.214.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
I20260924 16:05:22.583709 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:40951
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:40535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:23.002552 4046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:23.002967 4046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:23.003086 4046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:23.011811 4046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:23.011973 4046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:23.012112 4046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:05:23.024530 4046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:40535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:40951
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:23.026422 4046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:23.028105 4046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:23.045416 4052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:23.045346 4054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:23.045353 4051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:23.046497 4046 server_base.cc:1034] running on GCE node
I20260924 16:05:23.047344 4046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:23.048600 4046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:23.049891 4046 hybrid_clock.cc:648] HybridClock initialized: now 1790265923049814 us; error 56 us; skew 500 ppm
I20260924 16:05:23.050419 4046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:23 dist-test-slave-1gc3 krb5kdc[3938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265923, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:23.054796 4046 init.cc:377] Logged in from keytab as kudu/127.3.214.193@KRBTEST.COM (short username kudu)
I20260924 16:05:23.056828 4046 webserver.cc:460] Webserver started at http://127.3.214.193:43833/ using document root <none> and password file <none>
I20260924 16:05:23.057755 4046 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:23.057899 4046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:23.058333 4046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:23.062214 4046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "65bd6f63b821450cb483dbcb0e67912e"
format_stamp: "Formatted at 2026-09-24 16:05:23 on dist-test-slave-1gc3"
server_key: "be304c61173c459f285a50f6a71b7f09"
server_key_iv: "7dfe401546d0ebd5d18bd2d4b2742089"
server_key_version: "encryptionkey@0"
I20260924 16:05:23.063300 4046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "65bd6f63b821450cb483dbcb0e67912e"
format_stamp: "Formatted at 2026-09-24 16:05:23 on dist-test-slave-1gc3"
server_key: "be304c61173c459f285a50f6a71b7f09"
server_key_iv: "7dfe401546d0ebd5d18bd2d4b2742089"
server_key_version: "encryptionkey@0"
I20260924 16:05:23.070614 4046 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260924 16:05:23.075392 4062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:23.077354 4046 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:05:23.077615 4046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "65bd6f63b821450cb483dbcb0e67912e"
format_stamp: "Formatted at 2026-09-24 16:05:23 on dist-test-slave-1gc3"
server_key: "be304c61173c459f285a50f6a71b7f09"
server_key_iv: "7dfe401546d0ebd5d18bd2d4b2742089"
server_key_version: "encryptionkey@0"
I20260924 16:05:23.077888 4046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:23.104194 4046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:23.109557 4046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:23.109910 4046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:23.111220 4046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:23.112931 4046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:23.113018 4046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:23.113160 4046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:23.113216 4046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:23.152189 4046 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:38817
I20260924 16:05:23.152262 4174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:38817 every 8 connection(s)
I20260924 16:05:23.153972 4046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:05:23.161978 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 4046
I20260924 16:05:23.162285 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:05:23.162992 3931 external_mini_cluster.cc:1428] Setting key 941a664b3d166fb502707adc8d315523
Sep 24 16:05:23 dist-test-slave-1gc3 krb5kdc[3938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265923, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:23.176736 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:23.157253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:33299 (local address 127.3.214.254:40951)
0924 16:05:23.157871 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:23.157877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:23.159237 (+ 1360us) server_negotiation.cc:407] Connection header received
0924 16:05:23.161392 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:23.161398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:23.161496 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:23.161624 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:23.163527 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:23.164501 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:23.166163 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:23.166611 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:23.166733 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:23.170749 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:23.170754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:23.170757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:23.170794 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:23.173366 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:23.174255 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:23.174259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:23.174262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:23.174392 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:23.174859 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:23.174863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:23.174865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:23.175150 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:23.175288 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:23.176087 (+ 799us) server_negotiation.cc:299] Negotiation successful
0924 16:05:23.176343 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":311,"threads_started":1}
I20260924 16:05:23.177896 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:23.158025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40951 (local address 127.3.214.193:33299)
0924 16:05:23.159012 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:23.159071 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:23.161060 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:23.161890 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:23.161899 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:23.162614 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:23.163337 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:23.163364 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:23.164666 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:23.164672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:23.165976 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:23.165991 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:23.166853 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:23.166861 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:23.167129 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:23.168016 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:23.168053 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:23.170492 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:23.173550 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:23.173563 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:23.173581 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:23.174081 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:23.174512 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:23.174517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:23.174521 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:23.174735 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:23.175279 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:23.175288 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:23.175686 (+ 398us) client_negotiation.cc:769] Sending connection context
0924 16:05:23.175946 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 16:05:23.176565 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":354,"threads_started":1}
I20260924 16:05:23.179981 4175 heartbeater.cc:347] Connected to a master server at 127.3.214.254:40951
I20260924 16:05:23.180558 4175 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:23.181864 4175 heartbeater.cc:511] Master 127.3.214.254:40951 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.214.194@KRBTEST.COM; defaulting to no policy
I20260924 16:05:23.185146 3984 ts_manager.cc:194] Registered new tserver with Master: 65bd6f63b821450cb483dbcb0e67912e (127.3.214.193:38817)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.194@KRBTEST.COM" created.
I20260924 16:05:23.187456 3984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:33299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
I20260924 16:05:23.210430 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:40951
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:40535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:23.581210 4181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:23.581565 4181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:23.581696 4181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:23.589777 4181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:23.589943 4181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:23.590168 4181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:05:23.603977 4181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:40535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:40951
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:23.605443 4181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:23.607513 4181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:23.625555 4187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:23.625608 4186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:23.626462 4189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:23.626684 4181 server_base.cc:1034] running on GCE node
I20260924 16:05:23.627450 4181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:23.628556 4181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:23.629822 4181 hybrid_clock.cc:648] HybridClock initialized: now 1790265923629749 us; error 55 us; skew 500 ppm
I20260924 16:05:23.630347 4181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:23 dist-test-slave-1gc3 krb5kdc[3938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265923, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:23.633896 4181 init.cc:377] Logged in from keytab as kudu/127.3.214.194@KRBTEST.COM (short username kudu)
I20260924 16:05:23.635828 4181 webserver.cc:460] Webserver started at http://127.3.214.194:41467/ using document root <none> and password file <none>
I20260924 16:05:23.636646 4181 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:23.636901 4181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:23.637303 4181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:23.640758 4181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "eea2530ddf374543a31d880aa1bc5474"
format_stamp: "Formatted at 2026-09-24 16:05:23 on dist-test-slave-1gc3"
server_key: "fb48c4db3440a39a49feefe717b0dd09"
server_key_iv: "c8259b735e80e05068a7f91f4ba8a900"
server_key_version: "encryptionkey@0"
I20260924 16:05:23.641623 4181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "eea2530ddf374543a31d880aa1bc5474"
format_stamp: "Formatted at 2026-09-24 16:05:23 on dist-test-slave-1gc3"
server_key: "fb48c4db3440a39a49feefe717b0dd09"
server_key_iv: "c8259b735e80e05068a7f91f4ba8a900"
server_key_version: "encryptionkey@0"
I20260924 16:05:23.648238 4181 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260924 16:05:23.652945 4196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:23.654965 4181 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:05:23.655192 4181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "eea2530ddf374543a31d880aa1bc5474"
format_stamp: "Formatted at 2026-09-24 16:05:23 on dist-test-slave-1gc3"
server_key: "fb48c4db3440a39a49feefe717b0dd09"
server_key_iv: "c8259b735e80e05068a7f91f4ba8a900"
server_key_version: "encryptionkey@0"
I20260924 16:05:23.655416 4181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:23.678759 4181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:23.683722 4181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:23.684020 4181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:23.685227 4181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:23.686940 4181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:23.687033 4181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:23.687170 4181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:23.687227 4181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:23.724783 4181 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:34419
I20260924 16:05:23.724864 4311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:34419 every 8 connection(s)
I20260924 16:05:23.726245 4181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:05:23.732613 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 4181
I20260924 16:05:23.732874 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:05:23.733525 3931 external_mini_cluster.cc:1428] Setting key d162eef11e6a89b063d4c5cd3d9af723
Sep 24 16:05:23 dist-test-slave-1gc3 krb5kdc[3938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265923, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:23.751653 4314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:23.729622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:51899 (local address 127.3.214.254:40951)
0924 16:05:23.730214 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:23.730220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:23.731355 (+ 1135us) server_negotiation.cc:407] Connection header received
0924 16:05:23.734265 (+ 2910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:23.734270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:23.734394 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:23.734525 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:23.736775 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:23.737523 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:23.739232 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:23.739692 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:23.739803 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:23.744248 (+ 4445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:23.744256 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:23.744262 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:23.744308 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:23.747509 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:23.748639 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:23.748645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:23.748649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:23.748845 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:23.749355 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:23.749360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:23.749363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:23.749765 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:23.749940 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:23.750968 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0924 16:05:23.751234 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":276,"threads_started":1}
I20260924 16:05:23.752848 4315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:23.730279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40951 (local address 127.3.214.194:51899)
0924 16:05:23.731142 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:23.731194 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:23.733753 (+ 2559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:23.734952 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:23.734968 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:23.735865 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:23.736553 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:23.736577 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:23.737696 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:23.737700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:23.739063 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:23.739079 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:23.739953 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:23.739961 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:23.740230 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:23.740932 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:23.740963 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:23.743877 (+ 2914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:23.747739 (+ 3862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:23.747751 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:23.747774 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:23.748456 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:23.748971 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:23.748974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:23.748977 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:23.749217 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:23.749947 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:23.749957 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:23.750514 (+ 557us) client_negotiation.cc:769] Sending connection context
0924 16:05:23.750811 (+ 297us) client_negotiation.cc:240] Negotiation successful
0924 16:05:23.751390 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":311,"threads_started":1}
WARNING: no policy specified for kudu/127.3.214.195@KRBTEST.COM; defaulting to no policy
I20260924 16:05:23.754873 4312 heartbeater.cc:347] Connected to a master server at 127.3.214.254:40951
I20260924 16:05:23.755333 4312 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.195@KRBTEST.COM" created.
I20260924 16:05:23.756459 4312 heartbeater.cc:511] Master 127.3.214.254:40951 requested a full tablet report, sending...
I20260924 16:05:23.758903 3984 ts_manager.cc:194] Registered new tserver with Master: eea2530ddf374543a31d880aa1bc5474 (127.3.214.194:34419)
I20260924 16:05:23.759948 3984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:51899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
I20260924 16:05:23.782840 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:40951
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:40535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:05:24.192317 4175 heartbeater.cc:503] Master 127.3.214.254:40951 was elected leader, sending a full tablet report...
W20260924 16:05:24.202431 4318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:24.202858 4318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:24.203007 4318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:24.211460 4318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:24.211658 4318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:24.211840 4318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:05:24.223654 4318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:40535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:40951
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:24.225522 4318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:24.227442 4318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:24.244793 4323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:24.244863 4324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:24.244743 4326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:24.248087 4318 server_base.cc:1034] running on GCE node
I20260924 16:05:24.249082 4318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:24.250483 4318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:24.251878 4318 hybrid_clock.cc:648] HybridClock initialized: now 1790265924251805 us; error 57 us; skew 500 ppm
I20260924 16:05:24.252324 4318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:24 dist-test-slave-1gc3 krb5kdc[3938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265924, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:24.256520 4318 init.cc:377] Logged in from keytab as kudu/127.3.214.195@KRBTEST.COM (short username kudu)
I20260924 16:05:24.258729 4318 webserver.cc:460] Webserver started at http://127.3.214.195:36267/ using document root <none> and password file <none>
I20260924 16:05:24.259739 4318 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:24.259878 4318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:24.260269 4318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:24.263829 4318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "fa2ce541721c4bb19fd794e0745180e0"
format_stamp: "Formatted at 2026-09-24 16:05:24 on dist-test-slave-1gc3"
server_key: "d3510b97b480dad31344a5d51924da63"
server_key_iv: "bd9a456bbbc8acca1a84b30d9959aedc"
server_key_version: "encryptionkey@0"
I20260924 16:05:24.264734 4318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "fa2ce541721c4bb19fd794e0745180e0"
format_stamp: "Formatted at 2026-09-24 16:05:24 on dist-test-slave-1gc3"
server_key: "d3510b97b480dad31344a5d51924da63"
server_key_iv: "bd9a456bbbc8acca1a84b30d9959aedc"
server_key_version: "encryptionkey@0"
I20260924 16:05:24.271739 4318 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260924 16:05:24.276717 4333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:24.278717 4318 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:05:24.278982 4318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "fa2ce541721c4bb19fd794e0745180e0"
format_stamp: "Formatted at 2026-09-24 16:05:24 on dist-test-slave-1gc3"
server_key: "d3510b97b480dad31344a5d51924da63"
server_key_iv: "bd9a456bbbc8acca1a84b30d9959aedc"
server_key_version: "encryptionkey@0"
I20260924 16:05:24.279253 4318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:24.302007 4318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:24.306854 4318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:24.307175 4318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:24.308554 4318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:24.310422 4318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:24.310520 4318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:24.310658 4318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:24.310714 4318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:24.352753 4318 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:36351
I20260924 16:05:24.352782 4447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:36351 every 8 connection(s)
I20260924 16:05:24.354328 4318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:05:24.357517 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 4318
I20260924 16:05:24.357736 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:05:24.358374 3931 external_mini_cluster.cc:1428] Setting key f97b21bd9eaaf0f9396e8fff330ef049
Sep 24 16:05:24 dist-test-slave-1gc3 krb5kdc[3938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265924, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:24.381156 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:24.358212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:53611 (local address 127.3.214.254:40951)
0924 16:05:24.358895 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:24.358904 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:24.360272 (+ 1368us) server_negotiation.cc:407] Connection header received
0924 16:05:24.363250 (+ 2978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:24.363257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:24.363391 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:24.363505 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:24.365721 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:24.366639 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:24.368763 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:24.369235 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:24.369378 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:24.373794 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:24.373800 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:24.373804 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:24.373856 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:24.377256 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:24.378280 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:24.378287 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:24.378290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:24.378425 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:24.378922 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:24.378926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:24.378928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:24.379233 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:24.379383 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:24.380228 (+ 845us) server_negotiation.cc:299] Negotiation successful
0924 16:05:24.380495 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":287,"threads_started":1}
I20260924 16:05:24.382738 4451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:24.358896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40951 (local address 127.3.214.195:53611)
0924 16:05:24.359996 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:24.360079 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:24.362794 (+ 2715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:24.363901 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:24.363916 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:24.364666 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:24.365464 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:24.365487 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:24.366913 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:24.366918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:24.368528 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:24.368545 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:24.369514 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:24.369519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:24.369781 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:24.370633 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:24.370669 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:24.373361 (+ 2692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:24.377482 (+ 4121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:24.377494 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:24.377513 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:24.378151 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:24.378539 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:24.378545 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:24.378549 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:24.378779 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:24.379371 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:24.379381 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:24.379810 (+ 429us) client_negotiation.cc:769] Sending connection context
0924 16:05:24.380081 (+ 271us) client_negotiation.cc:240] Negotiation successful
0924 16:05:24.380945 (+ 864us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":381,"threads_started":1}
I20260924 16:05:24.385236 4448 heartbeater.cc:347] Connected to a master server at 127.3.214.254:40951
I20260924 16:05:24.385726 4448 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:24.386708 4448 heartbeater.cc:511] Master 127.3.214.254:40951 requested a full tablet report, sending...
I20260924 16:05:24.389137 3984 ts_manager.cc:194] Registered new tserver with Master: fa2ce541721c4bb19fd794e0745180e0 (127.3.214.195:36351)
I20260924 16:05:24.390455 3984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:53611
I20260924 16:05:24.390959 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 16:05:24 dist-test-slave-1gc3 krb5kdc[3938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265924, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 16:05:24 dist-test-slave-1gc3 krb5kdc[3938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265924, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.214.193@KRBTEST.COM
I20260924 16:05:24.449975 4456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:24.429896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49070 (local address 127.3.214.193:38817)
0924 16:05:24.431085 (+ 1189us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:24.431096 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:24.431137 (+ 41us) server_negotiation.cc:407] Connection header received
0924 16:05:24.431459 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:24.431471 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:24.431855 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:24.432093 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:24.433956 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:24.435401 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:24.436983 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:24.437715 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:24.437851 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:24.442731 (+ 4880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:24.442754 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:24.442779 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:24.442833 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:24.446494 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:24.447449 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:24.447458 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:24.447461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:24.447626 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:24.448072 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:24.448076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:24.448078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:24.448539 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:24.448872 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:24.449210 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 16:05:24.449546 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":966,"thread_start_us":776,"threads_started":1}
W20260924 16:05:24.451658 4090 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49070
Sep 24 16:05:24 dist-test-slave-1gc3 krb5kdc[3938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 16:05:24 dist-test-slave-1gc3 krb5kdc[3938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.193@KRBTEST.COM
I20260924 16:05:24.506601 4456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:24.490286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.3.214.193:38817)
0924 16:05:24.490625 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:24.490633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:24.491059 (+ 426us) server_negotiation.cc:407] Connection header received
0924 16:05:24.491252 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:24.491257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:24.491395 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:24.491507 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:24.493156 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:24.493977 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:24.495415 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:24.495890 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:24.496001 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:24.500336 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:24.500343 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:24.500347 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:24.500402 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:24.503415 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:24.504334 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:24.504338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:24.504341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:24.504525 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:24.504980 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:24.504984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:24.504986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:24.505301 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:24.505529 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:24.505935 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 16:05:24.506219 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 16:05:24.508495 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 4046
I20260924 16:05:24.523504 4456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:24.509815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49094 (local address 127.3.214.193:38817)
0924 16:05:24.510091 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:24.510134 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:24.510307 (+ 173us) server_negotiation.cc:407] Connection header received
0924 16:05:24.510491 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:24.510495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:24.510575 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:24.510665 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:24.511975 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:24.512837 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:24.514261 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:24.514724 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:24.514851 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:24.517505 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:24.517512 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:24.517516 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:24.517562 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:24.520582 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:24.521367 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:24.521372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:24.521375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:24.521535 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:24.521917 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:24.521921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:24.521923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:24.522288 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:24.522561 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:24.522860 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 16:05:24.523131 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 16:05:24.524628 4170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:24.623684 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 4046
I20260924 16:05:24.643734 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 4181
Sep 24 16:05:24 dist-test-slave-1gc3 krb5kdc[3938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.194@KRBTEST.COM
I20260924 16:05:24.663667 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:24.644945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39402 (local address 127.3.214.194:34419)
0924 16:05:24.645585 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:24.645592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:24.645626 (+ 34us) server_negotiation.cc:407] Connection header received
0924 16:05:24.645713 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:24.645721 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:24.645953 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:24.646177 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:24.647643 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:24.648751 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:24.650378 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:24.651039 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:24.651168 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:24.656071 (+ 4903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:24.656092 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:24.656113 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:24.656166 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:24.659880 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:24.661022 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:24.661032 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:24.661035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:24.661234 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:24.661709 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:24.661712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:24.661714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:24.662262 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:24.662557 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:24.663032 (+ 475us) server_negotiation.cc:299] Negotiation successful
0924 16:05:24.663339 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":316,"threads_started":1}
I20260924 16:05:24.665102 4307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:24.760998 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 4181
I20260924 16:05:24.777384 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 4318
Sep 24 16:05:24 dist-test-slave-1gc3 krb5kdc[3938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.195@KRBTEST.COM
I20260924 16:05:24.796962 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:24.778631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45444 (local address 127.3.214.195:36351)
0924 16:05:24.779271 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:24.779280 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:24.779319 (+ 39us) server_negotiation.cc:407] Connection header received
0924 16:05:24.779425 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:24.779433 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:24.779726 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:24.779882 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:24.781426 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:24.782699 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:24.784338 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:24.785085 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:24.785202 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:24.789797 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:24.789820 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:24.789846 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:24.789913 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:24.793458 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:24.794431 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:24.794441 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:24.794444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:24.794651 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:24.795094 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:24.795097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:24.795099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:24.795566 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:24.795833 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:24.796218 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 16:05:24.796508 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":270,"threads_started":1}
I20260924 16:05:24.798756 4443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:24.887894 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 4318
I20260924 16:05:24.904147 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 3952
I20260924 16:05:24.905764 4014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:25.029237 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 3952
2026-09-24T16:05:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4375 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:05:25 dist-test-slave-1gc3 krb5kdc[4470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:05:25 dist-test-slave-1gc3 krb5kdc[4470](info): set up 2 sockets
Sep 24 16:05:25 dist-test-slave-1gc3 krb5kdc[4470](info): commencing operation
krb5kdc: starting...
W20260924 16:05:26.055575 3931 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:05:26 dist-test-slave-1gc3 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:05:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:05:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:05:26.269784 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:35151
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:44295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:35151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:26.729122 4483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:26.729570 4483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:26.729707 4483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:26.740571 4483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:05:26.740782 4483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:26.740849 4483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:26.740948 4483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:05:26.741009 4483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:05:26.753173 4483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:44295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:35151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:35151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:26.755074 4483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:26.756718 4483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:26.771396 4488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:26.771257 4489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:26.771629 4491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:26.773221 4483 server_base.cc:1034] running on GCE node
I20260924 16:05:26.774364 4483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:26.776268 4483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:26.777518 4483 hybrid_clock.cc:648] HybridClock initialized: now 1790265926777451 us; error 55 us; skew 500 ppm
I20260924 16:05:26.777982 4483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:26 dist-test-slave-1gc3 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265926, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:26.789763 4483 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:05:26.792398 4483 webserver.cc:460] Webserver started at http://127.3.214.254:43671/ using document root <none> and password file <none>
I20260924 16:05:26.793627 4483 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:26.793802 4483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:26.794327 4483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:26.798867 4483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "54758fdb88784677b2c0ce4e4cdec48e"
format_stamp: "Formatted at 2026-09-24 16:05:26 on dist-test-slave-1gc3"
server_key: "382aa88a41bdaa7582a2f9c169acb049"
server_key_iv: "2b0bcc96903b4bd65213088febadd5f9"
server_key_version: "encryptionkey@0"
I20260924 16:05:26.799885 4483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "54758fdb88784677b2c0ce4e4cdec48e"
format_stamp: "Formatted at 2026-09-24 16:05:26 on dist-test-slave-1gc3"
server_key: "382aa88a41bdaa7582a2f9c169acb049"
server_key_iv: "2b0bcc96903b4bd65213088febadd5f9"
server_key_version: "encryptionkey@0"
I20260924 16:05:26.807046 4483 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260924 16:05:26.811898 4498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:26.813894 4483 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:05:26.814183 4483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "54758fdb88784677b2c0ce4e4cdec48e"
format_stamp: "Formatted at 2026-09-24 16:05:26 on dist-test-slave-1gc3"
server_key: "382aa88a41bdaa7582a2f9c169acb049"
server_key_iv: "2b0bcc96903b4bd65213088febadd5f9"
server_key_version: "encryptionkey@0"
I20260924 16:05:26.814457 4483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:26.850482 4483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:26.855996 4483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:26.856338 4483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:26.878937 4483 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:35151
I20260924 16:05:26.878981 4550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:35151 every 8 connection(s)
I20260924 16:05:26.880795 4483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:05:26.885869 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 4483
I20260924 16:05:26.885969 4551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:26.886168 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:05:26.886885 3931 external_mini_cluster.cc:1428] Setting key 120082a06b97805fa888d3eb43869a63
I20260924 16:05:26.903700 4551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54758fdb88784677b2c0ce4e4cdec48e: Bootstrap starting.
Sep 24 16:05:26 dist-test-slave-1gc3 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:26.907687 4551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54758fdb88784677b2c0ce4e4cdec48e: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:26.909219 4551 log.cc:824] T 00000000000000000000000000000000 P 54758fdb88784677b2c0ce4e4cdec48e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:26.913825 4551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54758fdb88784677b2c0ce4e4cdec48e: No bootstrap required, opened a new log
I20260924 16:05:26.918264 4554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:26.889580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54452 (local address 127.3.214.254:35151)
0924 16:05:26.890734 (+ 1154us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:26.890756 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:26.890829 (+ 73us) server_negotiation.cc:407] Connection header received
0924 16:05:26.893586 (+ 2757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:26.893629 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:26.894408 (+ 779us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:26.895011 (+ 603us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:26.897139 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:26.899651 (+ 2512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:26.901456 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:26.902674 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:26.902953 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:26.908058 (+ 5105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:26.908071 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:26.908097 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:26.908179 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:26.912465 (+ 4286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:26.913454 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:26.913464 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:26.913473 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:26.913723 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:26.914196 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:26.914202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:26.914204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:26.915101 (+ 897us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:26.915479 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:26.916140 (+ 661us) server_negotiation.cc:299] Negotiation successful
0924 16:05:26.916770 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":433,"threads_started":1}
I20260924 16:05:26.923313 4551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54758fdb88784677b2c0ce4e4cdec48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54758fdb88784677b2c0ce4e4cdec48e" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35151 } }
I20260924 16:05:26.923810 4551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54758fdb88784677b2c0ce4e4cdec48e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:26.923920 4551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54758fdb88784677b2c0ce4e4cdec48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54758fdb88784677b2c0ce4e4cdec48e, State: Initialized, Role: FOLLOWER
I20260924 16:05:26.924615 4551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54758fdb88784677b2c0ce4e4cdec48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54758fdb88784677b2c0ce4e4cdec48e" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35151 } }
I20260924 16:05:26.924911 4551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54758fdb88784677b2c0ce4e4cdec48e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:05:26.925072 4551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54758fdb88784677b2c0ce4e4cdec48e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:05:26.925292 4551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54758fdb88784677b2c0ce4e4cdec48e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:26.927810 4551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54758fdb88784677b2c0ce4e4cdec48e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54758fdb88784677b2c0ce4e4cdec48e" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35151 } }
I20260924 16:05:26.928375 4551 leader_election.cc:302] T 00000000000000000000000000000000 P 54758fdb88784677b2c0ce4e4cdec48e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54758fdb88784677b2c0ce4e4cdec48e; no voters:
I20260924 16:05:26.929062 4551 leader_election.cc:288] T 00000000000000000000000000000000 P 54758fdb88784677b2c0ce4e4cdec48e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:05:26.929277 4556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54758fdb88784677b2c0ce4e4cdec48e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:26.930302 4556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54758fdb88784677b2c0ce4e4cdec48e [term 1 LEADER]: Becoming Leader. State: Replica: 54758fdb88784677b2c0ce4e4cdec48e, State: Running, Role: LEADER
I20260924 16:05:26.931157 4556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54758fdb88784677b2c0ce4e4cdec48e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54758fdb88784677b2c0ce4e4cdec48e" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35151 } }
I20260924 16:05:26.931945 4551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54758fdb88784677b2c0ce4e4cdec48e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:05:26.935217 4557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54758fdb88784677b2c0ce4e4cdec48e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54758fdb88784677b2c0ce4e4cdec48e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54758fdb88784677b2c0ce4e4cdec48e" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35151 } } }
I20260924 16:05:26.935788 4557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54758fdb88784677b2c0ce4e4cdec48e [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:26.935246 4559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54758fdb88784677b2c0ce4e4cdec48e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54758fdb88784677b2c0ce4e4cdec48e. Latest consensus state: current_term: 1 leader_uuid: "54758fdb88784677b2c0ce4e4cdec48e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54758fdb88784677b2c0ce4e4cdec48e" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35151 } } }
I20260924 16:05:26.936187 4559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54758fdb88784677b2c0ce4e4cdec48e [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:26.938385 4563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:05:26.945647 4563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:05:26.959852 4563 catalog_manager.cc:1347] Generated new cluster ID: 63d360f9ba6847248190d62d01213cc0
I20260924 16:05:26.960026 4563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:05:26.971236 4563 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:05:26.973377 4563 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:05:26.983680 4563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54758fdb88784677b2c0ce4e4cdec48e: Generated new TSK 0
I20260924 16:05:26.985034 4563 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.214.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
I20260924 16:05:27.036072 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:35151
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:44295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:27.438359 4579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:27.438743 4579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:27.438859 4579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:27.447537 4579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:27.447698 4579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:27.447835 4579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:05:27.459837 4579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:44295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:35151
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:27.461645 4579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:27.463508 4579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:27.480489 4584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:27.480616 4585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:27.480684 4587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:27.481678 4579 server_base.cc:1034] running on GCE node
I20260924 16:05:27.482595 4579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:27.483968 4579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:27.485216 4579 hybrid_clock.cc:648] HybridClock initialized: now 1790265927485151 us; error 52 us; skew 500 ppm
I20260924 16:05:27.485690 4579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:27 dist-test-slave-1gc3 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265927, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:27.489864 4579 init.cc:377] Logged in from keytab as kudu/127.3.214.193@KRBTEST.COM (short username kudu)
I20260924 16:05:27.492007 4579 webserver.cc:460] Webserver started at http://127.3.214.193:44099/ using document root <none> and password file <none>
I20260924 16:05:27.492954 4579 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:27.493081 4579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:27.493475 4579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:27.497380 4579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "52fa1f9f9b60451a94e1a66da8d3e2fa"
format_stamp: "Formatted at 2026-09-24 16:05:27 on dist-test-slave-1gc3"
server_key: "42ab96b5bf46572aa0a76483cc64a220"
server_key_iv: "1266685d3308bc1bd1ac12c17cb4c51f"
server_key_version: "encryptionkey@0"
I20260924 16:05:27.498489 4579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "52fa1f9f9b60451a94e1a66da8d3e2fa"
format_stamp: "Formatted at 2026-09-24 16:05:27 on dist-test-slave-1gc3"
server_key: "42ab96b5bf46572aa0a76483cc64a220"
server_key_iv: "1266685d3308bc1bd1ac12c17cb4c51f"
server_key_version: "encryptionkey@0"
I20260924 16:05:27.505694 4579 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260924 16:05:27.510649 4594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:27.512712 4579 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:05:27.512966 4579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "52fa1f9f9b60451a94e1a66da8d3e2fa"
format_stamp: "Formatted at 2026-09-24 16:05:27 on dist-test-slave-1gc3"
server_key: "42ab96b5bf46572aa0a76483cc64a220"
server_key_iv: "1266685d3308bc1bd1ac12c17cb4c51f"
server_key_version: "encryptionkey@0"
I20260924 16:05:27.513226 4579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:27.536644 4579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:27.542390 4579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:27.542749 4579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:27.544044 4579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:27.545733 4579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:27.545846 4579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:27.545980 4579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:27.546036 4579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:27.584501 4579 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:43133
I20260924 16:05:27.584604 4707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:43133 every 8 connection(s)
I20260924 16:05:27.586180 4579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:05:27.590796 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 4579
I20260924 16:05:27.591030 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:05:27.591715 3931 external_mini_cluster.cc:1428] Setting key 6881bc9f956c7d008a8d4ea9e64e880a
Sep 24 16:05:27 dist-test-slave-1gc3 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265927, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:27.615263 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:27.589285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:34589 (local address 127.3.214.254:35151)
0924 16:05:27.589878 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:27.589887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:27.591872 (+ 1985us) server_negotiation.cc:407] Connection header received
0924 16:05:27.594247 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:27.594254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:27.594373 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:27.594523 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:27.597143 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:27.598019 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:27.600494 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:27.601200 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:27.601395 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:27.606302 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:27.606311 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:27.606316 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:27.606406 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:27.610580 (+ 4174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:27.611961 (+ 1381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:27.611969 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:27.611973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:27.612182 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:27.612818 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:27.612823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:27.612827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:27.613276 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:27.613461 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:27.614448 (+ 987us) server_negotiation.cc:299] Negotiation successful
0924 16:05:27.614803 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":266,"threads_started":1}
WARNING: no policy specified for kudu/127.3.214.194@KRBTEST.COM; defaulting to no policy
I20260924 16:05:27.616103 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:27.589774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:35151 (local address 127.3.214.193:34589)
0924 16:05:27.591579 (+ 1805us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:27.591644 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:27.593810 (+ 2166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:27.594919 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:27.594933 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:27.595867 (+ 934us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:27.596850 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:27.596882 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:27.598263 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:27.598268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:27.600233 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:27.600254 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:27.601628 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:27.601636 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:27.602006 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:27.603064 (+ 1058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:27.603102 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:27.605815 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:27.611036 (+ 5221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:27.611053 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:27.611080 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:27.611762 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:27.612389 (+ 627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:27.612393 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:27.612395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:27.612681 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:27.613498 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:27.613506 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:27.613940 (+ 434us) client_negotiation.cc:769] Sending connection context
0924 16:05:27.614278 (+ 338us) client_negotiation.cc:240] Negotiation successful
0924 16:05:27.614792 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1470,"thread_start_us":1188,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.194@KRBTEST.COM" created.
I20260924 16:05:27.618280 4708 heartbeater.cc:347] Connected to a master server at 127.3.214.254:35151
I20260924 16:05:27.618654 4708 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:27.619654 4708 heartbeater.cc:511] Master 127.3.214.254:35151 requested a full tablet report, sending...
I20260924 16:05:27.622237 4515 ts_manager.cc:194] Registered new tserver with Master: 52fa1f9f9b60451a94e1a66da8d3e2fa (127.3.214.193:43133)
I20260924 16:05:27.623845 4515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:34589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
I20260924 16:05:27.646247 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:35151
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:44295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:28.034509 4714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:28.034950 4714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:28.035113 4714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:28.043865 4714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:28.044021 4714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:28.044202 4714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:05:28.055680 4714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:44295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:35151
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:28.057497 4714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:28.059266 4714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:28.075237 4719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:28.075196 4723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:28.075721 4721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:28.076865 4714 server_base.cc:1034] running on GCE node
I20260924 16:05:28.077724 4714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:28.079108 4714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:28.080384 4714 hybrid_clock.cc:648] HybridClock initialized: now 1790265928080293 us; error 69 us; skew 500 ppm
I20260924 16:05:28.080824 4714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:28 dist-test-slave-1gc3 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265928, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:28.084824 4714 init.cc:377] Logged in from keytab as kudu/127.3.214.194@KRBTEST.COM (short username kudu)
I20260924 16:05:28.086894 4714 webserver.cc:460] Webserver started at http://127.3.214.194:33133/ using document root <none> and password file <none>
I20260924 16:05:28.087805 4714 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:28.087937 4714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:28.088300 4714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:28.092015 4714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "80a7cc5e79a84eae9fce3bce0e784ef3"
format_stamp: "Formatted at 2026-09-24 16:05:28 on dist-test-slave-1gc3"
server_key: "3f508aac0e8f26b6b89c9f51c4980284"
server_key_iv: "6d5bc8185b6a1198a79a67a987528962"
server_key_version: "encryptionkey@0"
I20260924 16:05:28.092993 4714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "80a7cc5e79a84eae9fce3bce0e784ef3"
format_stamp: "Formatted at 2026-09-24 16:05:28 on dist-test-slave-1gc3"
server_key: "3f508aac0e8f26b6b89c9f51c4980284"
server_key_iv: "6d5bc8185b6a1198a79a67a987528962"
server_key_version: "encryptionkey@0"
I20260924 16:05:28.100178 4714 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260924 16:05:28.105497 4730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:28.107583 4714 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:05:28.107847 4714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "80a7cc5e79a84eae9fce3bce0e784ef3"
format_stamp: "Formatted at 2026-09-24 16:05:28 on dist-test-slave-1gc3"
server_key: "3f508aac0e8f26b6b89c9f51c4980284"
server_key_iv: "6d5bc8185b6a1198a79a67a987528962"
server_key_version: "encryptionkey@0"
I20260924 16:05:28.108107 4714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:28.126863 4714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:28.132000 4714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:28.132341 4714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:28.133641 4714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:28.135452 4714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:28.135546 4714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:28.135692 4714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:28.135756 4714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:28.170889 4714 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:45303
I20260924 16:05:28.171012 4842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:45303 every 8 connection(s)
I20260924 16:05:28.172416 4714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:05:28.179785 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 4714
I20260924 16:05:28.179972 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:05:28.180632 3931 external_mini_cluster.cc:1428] Setting key 157aa08624a50c9c92b6b57beeb228ae
Sep 24 16:05:28 dist-test-slave-1gc3 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265928, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:28.199671 4845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:28.175971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:33563 (local address 127.3.214.254:35151)
0924 16:05:28.176602 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:28.176608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:28.177636 (+ 1028us) server_negotiation.cc:407] Connection header received
0924 16:05:28.180151 (+ 2515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:28.180159 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:28.180295 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:28.180456 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:28.182709 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.183612 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.186325 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.186819 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.186931 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:28.192263 (+ 5332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:28.192270 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:28.192274 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:28.192324 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:28.195264 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:28.196461 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:28.196467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:28.196470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:28.196629 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:28.197264 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:28.197268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:28.197271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:28.197784 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:28.197935 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:28.199093 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0924 16:05:28.199349 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":369,"threads_started":1}
I20260924 16:05:28.201263 4846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:28.176358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:35151 (local address 127.3.214.194:33563)
0924 16:05:28.177393 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:28.177459 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:28.179590 (+ 2131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:28.180835 (+ 1245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:28.180847 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:28.181590 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:28.182494 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:28.182517 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.183841 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.183847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:28.186089 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:28.186142 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.187114 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.187121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:28.187500 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:28.188429 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:28.188467 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:28.191836 (+ 3369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:28.195496 (+ 3660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:28.195509 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:28.195533 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:28.196250 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:28.196759 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:28.196763 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:28.196767 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:28.197085 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:28.197951 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:28.197962 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:28.198579 (+ 617us) client_negotiation.cc:769] Sending connection context
0924 16:05:28.198964 (+ 385us) client_negotiation.cc:240] Negotiation successful
0924 16:05:28.199636 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":407,"threads_started":1}
WARNING: no policy specified for kudu/127.3.214.195@KRBTEST.COM; defaulting to no policy
I20260924 16:05:28.203493 4843 heartbeater.cc:347] Connected to a master server at 127.3.214.254:35151
I20260924 16:05:28.203889 4843 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.195@KRBTEST.COM" created.
I20260924 16:05:28.204878 4843 heartbeater.cc:511] Master 127.3.214.254:35151 requested a full tablet report, sending...
I20260924 16:05:28.207214 4515 ts_manager.cc:194] Registered new tserver with Master: 80a7cc5e79a84eae9fce3bce0e784ef3 (127.3.214.194:45303)
I20260924 16:05:28.208132 4515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:33563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
I20260924 16:05:28.230579 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:35151
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:44295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:28.585865 4849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:28.586239 4849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:28.586412 4849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:28.596140 4849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:28.596313 4849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:28.596515 4849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:05:28.608129 4849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:44295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:35151
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:28.609560 4849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:28.611014 4849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:28.626356 4857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:28.626473 4854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:28.626403 4855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:28.627476 4849 server_base.cc:1034] running on GCE node
I20260924 16:05:28.628123 4708 heartbeater.cc:503] Master 127.3.214.254:35151 was elected leader, sending a full tablet report...
I20260924 16:05:28.628258 4849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:28.629526 4849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:28.630916 4849 hybrid_clock.cc:648] HybridClock initialized: now 1790265928630824 us; error 86 us; skew 500 ppm
I20260924 16:05:28.631309 4849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:28 dist-test-slave-1gc3 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265928, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:28.634763 4849 init.cc:377] Logged in from keytab as kudu/127.3.214.195@KRBTEST.COM (short username kudu)
I20260924 16:05:28.636606 4849 webserver.cc:460] Webserver started at http://127.3.214.195:42223/ using document root <none> and password file <none>
I20260924 16:05:28.637413 4849 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:28.637533 4849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:28.637851 4849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:28.641146 4849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "9e5e4a444baa4fc7925fd79830548852"
format_stamp: "Formatted at 2026-09-24 16:05:28 on dist-test-slave-1gc3"
server_key: "bc2f0dee546cc2870bd29a2d9e6fd020"
server_key_iv: "471c23b459094204a97635fb3f1cb192"
server_key_version: "encryptionkey@0"
I20260924 16:05:28.642020 4849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "9e5e4a444baa4fc7925fd79830548852"
format_stamp: "Formatted at 2026-09-24 16:05:28 on dist-test-slave-1gc3"
server_key: "bc2f0dee546cc2870bd29a2d9e6fd020"
server_key_iv: "471c23b459094204a97635fb3f1cb192"
server_key_version: "encryptionkey@0"
I20260924 16:05:28.648278 4849 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 16:05:28.652699 4865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:28.654783 4849 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:05:28.654995 4849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "9e5e4a444baa4fc7925fd79830548852"
format_stamp: "Formatted at 2026-09-24 16:05:28 on dist-test-slave-1gc3"
server_key: "bc2f0dee546cc2870bd29a2d9e6fd020"
server_key_iv: "471c23b459094204a97635fb3f1cb192"
server_key_version: "encryptionkey@0"
I20260924 16:05:28.655203 4849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:28.676362 4849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:28.680922 4849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:28.681187 4849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:28.682379 4849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:28.683979 4849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:28.684067 4849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:28.684170 4849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:28.684218 4849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:28.722759 4849 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:35945
I20260924 16:05:28.722882 4977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:35945 every 8 connection(s)
I20260924 16:05:28.724200 4849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:05:28.734122 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 4849
I20260924 16:05:28.734402 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:05:28.735073 3931 external_mini_cluster.cc:1428] Setting key 960527c47e46e8ad21f8b007b445fa0a
Sep 24 16:05:28 dist-test-slave-1gc3 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265928, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:28.752357 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:28.727496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:49593 (local address 127.3.214.254:35151)
0924 16:05:28.728113 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:28.728121 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:28.729496 (+ 1375us) server_negotiation.cc:407] Connection header received
0924 16:05:28.731748 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:28.731762 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:28.731910 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:28.732049 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:28.734123 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.735450 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.737491 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.738011 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.738179 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:28.742642 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:28.742651 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:28.742656 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:28.742720 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:28.746887 (+ 4167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:28.748117 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:28.748123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:28.748126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:28.748314 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:28.748950 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:28.748955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:28.748957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:28.750026 (+ 1069us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:28.750363 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:28.751431 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0924 16:05:28.751762 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":259,"threads_started":1}
I20260924 16:05:28.753710 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:28.728250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:35151 (local address 127.3.214.195:49593)
0924 16:05:28.729279 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:28.729338 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:28.731287 (+ 1949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:28.732360 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:28.732370 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:28.733132 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:28.733851 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:28.733873 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.735827 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.735832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:28.737273 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:28.737292 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.738360 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.738368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:28.738743 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:28.739518 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:28.739551 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:28.742225 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:28.747170 (+ 4945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:28.747185 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:28.747210 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:28.747909 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:28.748491 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:28.748496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:28.748499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:28.748786 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:28.750353 (+ 1567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:28.750366 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:28.750925 (+ 559us) client_negotiation.cc:769] Sending connection context
0924 16:05:28.751277 (+ 352us) client_negotiation.cc:240] Negotiation successful
0924 16:05:28.752138 (+ 861us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":389,"threads_started":1}
I20260924 16:05:28.755504 4978 heartbeater.cc:347] Connected to a master server at 127.3.214.254:35151
I20260924 16:05:28.755964 4978 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:28.756961 4978 heartbeater.cc:511] Master 127.3.214.254:35151 requested a full tablet report, sending...
I20260924 16:05:28.759093 4515 ts_manager.cc:194] Registered new tserver with Master: 9e5e4a444baa4fc7925fd79830548852 (127.3.214.195:35945)
I20260924 16:05:28.760066 4515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:49593
I20260924 16:05:28.765501 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:05:28.789284 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:28.774159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54464 (local address 127.3.214.254:35151)
0924 16:05:28.774411 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:28.774417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:28.774898 (+ 481us) server_negotiation.cc:407] Connection header received
0924 16:05:28.775106 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:28.775111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:28.775200 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:28.775337 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:28.776989 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.777899 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.779305 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.779854 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.779979 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:28.782785 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:28.782791 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:28.782795 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:28.782843 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:28.785898 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:28.786960 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:28.786965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:28.786967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:28.787129 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:28.787563 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:28.787566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:28.787568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:28.787970 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:28.788182 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:28.788717 (+ 535us) server_negotiation.cc:299] Negotiation successful
0924 16:05:28.788963 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 16:05:28.798237 4515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:05:28.806538 4515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:05:28.839661 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:28.822565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:45303 (local address 127.0.0.1:36070)
0924 16:05:28.823482 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:28.823529 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:28.823767 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:28.825104 (+ 1337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:28.825113 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:28.825157 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:28.825640 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:28.825655 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.833944 (+ 8289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.833949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:28.836462 (+ 2513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:28.836476 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.838439 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.838446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:28.838658 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:28.838726 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 16:05:28.838893 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 16:05:28.839141 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":5,"thread_start_us":1341,"threads_started":1}
I20260924 16:05:28.840513 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:28.822759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36070 (local address 127.3.214.194:45303)
0924 16:05:28.824374 (+ 1615us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:28.824384 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:28.824427 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:05:28.824536 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:28.824545 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:28.824834 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:28.831796 (+ 6962us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:28.832185 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.833753 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.836707 (+ 2954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.838140 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.838367 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:28.838623 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:28.838990 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 16:05:28.839195 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1462,"thread_start_us":1361,"threads_started":1}
I20260924 16:05:28.841351 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:28.823102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:35945 (local address 127.0.0.1:54482)
0924 16:05:28.830623 (+ 7521us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:28.830653 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:28.830828 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:28.831988 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:28.831994 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:28.832063 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:28.832537 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:28.832552 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.834940 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.834948 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:28.837416 (+ 2468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:28.837438 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.840244 (+ 2806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.840254 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:28.840458 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:28.840490 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 16:05:28.840584 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 16:05:28.840955 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7340,"mutex_wait_us":33,"thread_start_us":7050,"threads_started":1}
I20260924 16:05:28.841777 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:28.823708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54482 (local address 127.3.214.195:35945)
0924 16:05:28.831078 (+ 7370us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:28.831093 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:28.831144 (+ 51us) server_negotiation.cc:407] Connection header received
0924 16:05:28.831305 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:28.831315 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:28.831694 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:28.832803 (+ 1109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:28.833261 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.834733 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.837652 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.839338 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.839478 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:28.839751 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:28.841183 (+ 1432us) server_negotiation.cc:299] Negotiation successful
0924 16:05:28.841420 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7064,"thread_start_us":7223,"threads_started":1}
I20260924 16:05:28.842696 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:28.826201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49448 (local address 127.3.214.193:43133)
0924 16:05:28.832466 (+ 6265us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:28.832475 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:28.832505 (+ 30us) server_negotiation.cc:407] Connection header received
0924 16:05:28.832748 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:28.832757 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:28.833057 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:28.833293 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:28.834274 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.835920 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.838245 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.839712 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.839856 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:28.840123 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:28.842064 (+ 1941us) server_negotiation.cc:299] Negotiation successful
0924 16:05:28.842318 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6092,"thread_start_us":5911,"threads_started":1}
I20260924 16:05:28.843518 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:28.824977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:43133 (local address 127.0.0.1:49448)
0924 16:05:28.831725 (+ 6748us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:28.831748 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:28.831876 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:28.833299 (+ 1423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:28.833304 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:28.833333 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:28.833726 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:28.833737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.836094 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.836101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:28.838021 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:28.838037 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.840306 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.840315 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:28.840531 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:28.840574 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 16:05:28.840682 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 16:05:28.843095 (+ 2413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6613,"mutex_wait_us":2187,"thread_start_us":6470,"threads_started":1}
I20260924 16:05:28.845461 4778 tablet_service.cc:1467] Processing CreateTablet for tablet c9464b2bc85e4800ba207762c097677f (DEFAULT_TABLE table=test-table [id=e961338944f0458aae4125d66f7009c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:28.846223 4913 tablet_service.cc:1467] Processing CreateTablet for tablet c9464b2bc85e4800ba207762c097677f (DEFAULT_TABLE table=test-table [id=e961338944f0458aae4125d66f7009c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:28.847177 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9464b2bc85e4800ba207762c097677f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:28.848088 4913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9464b2bc85e4800ba207762c097677f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:28.848006 4643 tablet_service.cc:1467] Processing CreateTablet for tablet c9464b2bc85e4800ba207762c097677f (DEFAULT_TABLE table=test-table [id=e961338944f0458aae4125d66f7009c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:28.849690 4643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9464b2bc85e4800ba207762c097677f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:28.859824 4997 tablet_bootstrap.cc:492] T c9464b2bc85e4800ba207762c097677f P 80a7cc5e79a84eae9fce3bce0e784ef3: Bootstrap starting.
I20260924 16:05:28.863874 4997 tablet_bootstrap.cc:654] T c9464b2bc85e4800ba207762c097677f P 80a7cc5e79a84eae9fce3bce0e784ef3: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:28.865444 4997 log.cc:824] T c9464b2bc85e4800ba207762c097677f P 80a7cc5e79a84eae9fce3bce0e784ef3: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:28.865837 4998 tablet_bootstrap.cc:492] T c9464b2bc85e4800ba207762c097677f P 52fa1f9f9b60451a94e1a66da8d3e2fa: Bootstrap starting.
I20260924 16:05:28.870543 4998 tablet_bootstrap.cc:654] T c9464b2bc85e4800ba207762c097677f P 52fa1f9f9b60451a94e1a66da8d3e2fa: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:28.871770 4999 tablet_bootstrap.cc:492] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852: Bootstrap starting.
I20260924 16:05:28.872268 4998 log.cc:824] T c9464b2bc85e4800ba207762c097677f P 52fa1f9f9b60451a94e1a66da8d3e2fa: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:28.876380 4999 tablet_bootstrap.cc:654] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:28.876812 4997 tablet_bootstrap.cc:492] T c9464b2bc85e4800ba207762c097677f P 80a7cc5e79a84eae9fce3bce0e784ef3: No bootstrap required, opened a new log
I20260924 16:05:28.877175 4997 ts_tablet_manager.cc:1397] T c9464b2bc85e4800ba207762c097677f P 80a7cc5e79a84eae9fce3bce0e784ef3: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.007s
I20260924 16:05:28.877758 4999 log.cc:824] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:28.877830 4998 tablet_bootstrap.cc:492] T c9464b2bc85e4800ba207762c097677f P 52fa1f9f9b60451a94e1a66da8d3e2fa: No bootstrap required, opened a new log
I20260924 16:05:28.878172 4998 ts_tablet_manager.cc:1397] T c9464b2bc85e4800ba207762c097677f P 52fa1f9f9b60451a94e1a66da8d3e2fa: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260924 16:05:28.888350 4997 raft_consensus.cc:348] T c9464b2bc85e4800ba207762c097677f P 80a7cc5e79a84eae9fce3bce0e784ef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a7cc5e79a84eae9fce3bce0e784ef3" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45303 } } peers { permanent_uuid: "9e5e4a444baa4fc7925fd79830548852" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35945 } } peers { permanent_uuid: "52fa1f9f9b60451a94e1a66da8d3e2fa" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43133 } }
I20260924 16:05:28.888935 4997 raft_consensus.cc:374] T c9464b2bc85e4800ba207762c097677f P 80a7cc5e79a84eae9fce3bce0e784ef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:28.889026 4997 raft_consensus.cc:729] T c9464b2bc85e4800ba207762c097677f P 80a7cc5e79a84eae9fce3bce0e784ef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a7cc5e79a84eae9fce3bce0e784ef3, State: Initialized, Role: FOLLOWER
I20260924 16:05:28.888772 4998 raft_consensus.cc:348] T c9464b2bc85e4800ba207762c097677f P 52fa1f9f9b60451a94e1a66da8d3e2fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a7cc5e79a84eae9fce3bce0e784ef3" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45303 } } peers { permanent_uuid: "9e5e4a444baa4fc7925fd79830548852" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35945 } } peers { permanent_uuid: "52fa1f9f9b60451a94e1a66da8d3e2fa" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43133 } }
I20260924 16:05:28.889211 4998 raft_consensus.cc:374] T c9464b2bc85e4800ba207762c097677f P 52fa1f9f9b60451a94e1a66da8d3e2fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:28.889294 4998 raft_consensus.cc:729] T c9464b2bc85e4800ba207762c097677f P 52fa1f9f9b60451a94e1a66da8d3e2fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52fa1f9f9b60451a94e1a66da8d3e2fa, State: Initialized, Role: FOLLOWER
I20260924 16:05:28.889942 4998 consensus_queue.cc:260] T c9464b2bc85e4800ba207762c097677f P 52fa1f9f9b60451a94e1a66da8d3e2fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a7cc5e79a84eae9fce3bce0e784ef3" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45303 } } peers { permanent_uuid: "9e5e4a444baa4fc7925fd79830548852" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35945 } } peers { permanent_uuid: "52fa1f9f9b60451a94e1a66da8d3e2fa" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43133 } }
I20260924 16:05:28.889937 4997 consensus_queue.cc:260] T c9464b2bc85e4800ba207762c097677f P 80a7cc5e79a84eae9fce3bce0e784ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a7cc5e79a84eae9fce3bce0e784ef3" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45303 } } peers { permanent_uuid: "9e5e4a444baa4fc7925fd79830548852" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35945 } } peers { permanent_uuid: "52fa1f9f9b60451a94e1a66da8d3e2fa" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43133 } }
I20260924 16:05:28.896580 4843 heartbeater.cc:503] Master 127.3.214.254:35151 was elected leader, sending a full tablet report...
I20260924 16:05:28.897411 4997 ts_tablet_manager.cc:1428] T c9464b2bc85e4800ba207762c097677f P 80a7cc5e79a84eae9fce3bce0e784ef3: Time spent starting tablet: real 0.020s user 0.012s sys 0.008s
I20260924 16:05:28.897671 4998 ts_tablet_manager.cc:1428] T c9464b2bc85e4800ba207762c097677f P 52fa1f9f9b60451a94e1a66da8d3e2fa: Time spent starting tablet: real 0.019s user 0.010s sys 0.008s
I20260924 16:05:28.900339 4999 tablet_bootstrap.cc:492] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852: No bootstrap required, opened a new log
I20260924 16:05:28.900656 4999 ts_tablet_manager.cc:1397] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.005s
I20260924 16:05:28.909245 4999 raft_consensus.cc:348] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a7cc5e79a84eae9fce3bce0e784ef3" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45303 } } peers { permanent_uuid: "9e5e4a444baa4fc7925fd79830548852" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35945 } } peers { permanent_uuid: "52fa1f9f9b60451a94e1a66da8d3e2fa" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43133 } }
I20260924 16:05:28.909708 4999 raft_consensus.cc:374] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:28.909809 4999 raft_consensus.cc:729] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e5e4a444baa4fc7925fd79830548852, State: Initialized, Role: FOLLOWER
I20260924 16:05:28.910480 4999 consensus_queue.cc:260] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a7cc5e79a84eae9fce3bce0e784ef3" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45303 } } peers { permanent_uuid: "9e5e4a444baa4fc7925fd79830548852" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35945 } } peers { permanent_uuid: "52fa1f9f9b60451a94e1a66da8d3e2fa" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43133 } }
I20260924 16:05:28.911798 4978 heartbeater.cc:503] Master 127.3.214.254:35151 was elected leader, sending a full tablet report...
I20260924 16:05:28.912523 4999 ts_tablet_manager.cc:1428] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260924 16:05:28.924795 4844 tablet.cc:2369] T c9464b2bc85e4800ba207762c097677f P 80a7cc5e79a84eae9fce3bce0e784ef3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:05:28.925021 5005 raft_consensus.cc:482] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:05:28.925413 5005 raft_consensus.cc:504] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a7cc5e79a84eae9fce3bce0e784ef3" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45303 } } peers { permanent_uuid: "9e5e4a444baa4fc7925fd79830548852" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35945 } } peers { permanent_uuid: "52fa1f9f9b60451a94e1a66da8d3e2fa" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43133 } }
I20260924 16:05:28.927563 5005 leader_election.cc:288] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80a7cc5e79a84eae9fce3bce0e784ef3 (127.3.214.194:45303), 52fa1f9f9b60451a94e1a66da8d3e2fa (127.3.214.193:43133)
I20260924 16:05:28.934880 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:28.928276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:38689 (local address 127.3.214.194:45303)
0924 16:05:28.928518 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:28.928523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:28.928542 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:05:28.928635 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:28.928641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:28.928732 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:28.928842 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:28.929380 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.930441 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.932494 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.933794 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.933929 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:28.933992 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:28.934473 (+ 481us) server_negotiation.cc:299] Negotiation successful
0924 16:05:28.934609 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 16:05:28.934932 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:28.927932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:45303 (local address 127.3.214.195:38689)
0924 16:05:28.928183 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:28.928207 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:28.928365 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:28.928925 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:28.928931 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:28.928981 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:28.929208 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:28.929222 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.930589 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.930595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:28.932309 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:28.932326 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.933966 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.933973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:28.934223 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:28.934267 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 16:05:28.934381 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 16:05:28.934587 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 16:05:28.936036 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:28.928611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:52097 (local address 127.3.214.193:43133)
0924 16:05:28.928806 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:28.928813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:28.929189 (+ 376us) server_negotiation.cc:407] Connection header received
0924 16:05:28.929351 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:28.929357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:28.929430 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:28.929500 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:28.930231 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.931288 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.933669 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.934947 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.935115 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:28.935193 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:28.935728 (+ 535us) server_negotiation.cc:299] Negotiation successful
0924 16:05:28.935848 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:05:28.936230 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:28.928305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:43133 (local address 127.3.214.195:52097)
0924 16:05:28.929080 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:28.929096 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:28.929208 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:28.929558 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:28.929563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:28.929590 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:28.930005 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:28.930017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.931476 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.931480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:28.933483 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:28.933499 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.935367 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.935373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:28.935538 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:28.935568 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 16:05:28.935662 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 16:05:28.935864 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":500,"threads_started":1}
I20260924 16:05:28.936480 4798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9464b2bc85e4800ba207762c097677f" candidate_uuid: "9e5e4a444baa4fc7925fd79830548852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a7cc5e79a84eae9fce3bce0e784ef3" is_pre_election: true
I20260924 16:05:28.937187 4798 raft_consensus.cc:2454] T c9464b2bc85e4800ba207762c097677f P 80a7cc5e79a84eae9fce3bce0e784ef3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e5e4a444baa4fc7925fd79830548852 in term 0.
I20260924 16:05:28.937361 4663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9464b2bc85e4800ba207762c097677f" candidate_uuid: "9e5e4a444baa4fc7925fd79830548852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52fa1f9f9b60451a94e1a66da8d3e2fa" is_pre_election: true
I20260924 16:05:28.937987 4663 raft_consensus.cc:2454] T c9464b2bc85e4800ba207762c097677f P 52fa1f9f9b60451a94e1a66da8d3e2fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e5e4a444baa4fc7925fd79830548852 in term 0.
I20260924 16:05:28.938011 4868 leader_election.cc:302] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a7cc5e79a84eae9fce3bce0e784ef3, 9e5e4a444baa4fc7925fd79830548852; no voters:
I20260924 16:05:28.938548 5005 raft_consensus.cc:2781] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:05:28.938726 5005 raft_consensus.cc:482] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:05:28.938846 5005 raft_consensus.cc:3037] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:28.941299 5005 raft_consensus.cc:504] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a7cc5e79a84eae9fce3bce0e784ef3" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45303 } } peers { permanent_uuid: "9e5e4a444baa4fc7925fd79830548852" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35945 } } peers { permanent_uuid: "52fa1f9f9b60451a94e1a66da8d3e2fa" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43133 } }
I20260924 16:05:28.941988 5005 leader_election.cc:288] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852 [CANDIDATE]: Term 1 election: Requested vote from peers 80a7cc5e79a84eae9fce3bce0e784ef3 (127.3.214.194:45303), 52fa1f9f9b60451a94e1a66da8d3e2fa (127.3.214.193:43133)
I20260924 16:05:28.942698 4798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9464b2bc85e4800ba207762c097677f" candidate_uuid: "9e5e4a444baa4fc7925fd79830548852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a7cc5e79a84eae9fce3bce0e784ef3"
I20260924 16:05:28.942895 4663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9464b2bc85e4800ba207762c097677f" candidate_uuid: "9e5e4a444baa4fc7925fd79830548852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52fa1f9f9b60451a94e1a66da8d3e2fa"
I20260924 16:05:28.942962 4798 raft_consensus.cc:3037] T c9464b2bc85e4800ba207762c097677f P 80a7cc5e79a84eae9fce3bce0e784ef3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:28.943127 4663 raft_consensus.cc:3037] T c9464b2bc85e4800ba207762c097677f P 52fa1f9f9b60451a94e1a66da8d3e2fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:28.945714 4798 raft_consensus.cc:2454] T c9464b2bc85e4800ba207762c097677f P 80a7cc5e79a84eae9fce3bce0e784ef3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e5e4a444baa4fc7925fd79830548852 in term 1.
I20260924 16:05:28.945904 4663 raft_consensus.cc:2454] T c9464b2bc85e4800ba207762c097677f P 52fa1f9f9b60451a94e1a66da8d3e2fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e5e4a444baa4fc7925fd79830548852 in term 1.
I20260924 16:05:28.946252 4868 leader_election.cc:302] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a7cc5e79a84eae9fce3bce0e784ef3, 9e5e4a444baa4fc7925fd79830548852; no voters:
I20260924 16:05:28.946671 5005 raft_consensus.cc:2781] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:28.947201 5005 raft_consensus.cc:686] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852 [term 1 LEADER]: Becoming Leader. State: Replica: 9e5e4a444baa4fc7925fd79830548852, State: Running, Role: LEADER
I20260924 16:05:28.947744 5005 consensus_queue.cc:237] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a7cc5e79a84eae9fce3bce0e784ef3" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45303 } } peers { permanent_uuid: "9e5e4a444baa4fc7925fd79830548852" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35945 } } peers { permanent_uuid: "52fa1f9f9b60451a94e1a66da8d3e2fa" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43133 } }
I20260924 16:05:28.953157 4515 catalog_manager.cc:5495] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e5e4a444baa4fc7925fd79830548852 (127.3.214.195). New cstate: current_term: 1 leader_uuid: "9e5e4a444baa4fc7925fd79830548852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a7cc5e79a84eae9fce3bce0e784ef3" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e5e4a444baa4fc7925fd79830548852" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52fa1f9f9b60451a94e1a66da8d3e2fa" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43133 } health_report { overall_health: UNKNOWN } } }
W20260924 16:05:28.976238 4979 tablet.cc:2369] T c9464b2bc85e4800ba207762c097677f P 9e5e4a444baa4fc7925fd79830548852: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 16:05:28 dist-test-slave-1gc3 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.193@KRBTEST.COM
I20260924 16:05:29.002372 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:28.983818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49458 (local address 127.3.214.193:43133)
0924 16:05:28.984077 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:28.984085 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:28.984725 (+ 640us) server_negotiation.cc:407] Connection header received
0924 16:05:28.984905 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:28.984909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:28.985012 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:28.985117 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:28.986915 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.987799 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.989308 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:28.989785 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:28.989895 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:28.994768 (+ 4873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:28.994790 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:28.994815 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:28.994869 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:28.998476 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:28.999496 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:28.999506 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:28.999509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:28.999741 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:29.000260 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:29.000264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:29.000267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:29.000834 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:29.001158 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:29.001576 (+ 418us) server_negotiation.cc:299] Negotiation successful
0924 16:05:29.001912 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Sep 24 16:05:29 dist-test-slave-1gc3 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265929, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 16:05:29 dist-test-slave-1gc3 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265929, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.214.193@KRBTEST.COM
I20260924 16:05:29.057052 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:29.040035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49460 (local address 127.3.214.193:43133)
0924 16:05:29.040321 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:29.040328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:29.040726 (+ 398us) server_negotiation.cc:407] Connection header received
0924 16:05:29.040872 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:29.040877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:29.040971 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:29.041105 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:29.042809 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:29.043795 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:29.045364 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:29.045853 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:29.045961 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:29.050464 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:29.050470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:29.050474 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:29.050528 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:29.054005 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:29.054981 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:29.054986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:29.054989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:29.055166 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:29.055586 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:29.055589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:29.055591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:29.055922 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:29.056185 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:29.056427 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 16:05:29.056674 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260924 16:05:29.089330 4709 tablet.cc:2369] T c9464b2bc85e4800ba207762c097677f P 52fa1f9f9b60451a94e1a66da8d3e2fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 16:05:29 dist-test-slave-1gc3 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 16:05:29 dist-test-slave-1gc3 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.193@KRBTEST.COM
I20260924 16:05:29.115371 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:29.098188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49476 (local address 127.3.214.193:43133)
0924 16:05:29.098498 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:29.098504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:29.098719 (+ 215us) server_negotiation.cc:407] Connection header received
0924 16:05:29.098903 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:29.098909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:29.098995 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:29.099106 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:29.100770 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:29.101557 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:29.103159 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:29.103698 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:29.103828 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:29.109166 (+ 5338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:29.109173 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:29.109177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:29.109239 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:29.112072 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:29.113012 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:29.113017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:29.113019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:29.113188 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:29.113696 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:29.113700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:29.113702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:29.114149 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:29.114488 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:29.114782 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 16:05:29.115027 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
W20260924 16:05:29.117772 4622 tablet_service.cc:2908] Rejecting scan request for tablet c9464b2bc85e4800ba207762c097677f: Uninitialized: clean time has not yet been initialized
I20260924 16:05:29.122309 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 4579
I20260924 16:05:29.138352 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:29.123499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49492 (local address 127.3.214.193:43133)
0924 16:05:29.123722 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:29.123727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:29.124074 (+ 347us) server_negotiation.cc:407] Connection header received
0924 16:05:29.124251 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:29.124256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:29.124352 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:29.124459 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:29.125784 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:29.126758 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:29.128216 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:29.128756 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:29.128894 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:29.131713 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:29.131721 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:29.131727 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:29.131790 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:29.135042 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:29.136007 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:29.136012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:29.136015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:29.136188 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:29.136724 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:29.136728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:29.136730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:29.137110 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:29.137387 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:29.137770 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 16:05:29.138011 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 16:05:29.139243 4703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:29.236725 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 4579
I20260924 16:05:29.254742 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 4714
Sep 24 16:05:29 dist-test-slave-1gc3 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.194@KRBTEST.COM
I20260924 16:05:29.272751 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:29.255822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36080 (local address 127.3.214.194:45303)
0924 16:05:29.256118 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:29.256124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:29.256171 (+ 47us) server_negotiation.cc:407] Connection header received
0924 16:05:29.256250 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:29.256255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:29.256342 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:29.256447 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:29.257750 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:29.258915 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:29.260398 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:29.260883 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:29.261007 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:29.265600 (+ 4593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:29.265627 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:29.265652 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:29.265735 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:29.269195 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:29.270155 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:29.270164 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:29.270167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:29.270343 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:29.270852 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:29.270856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:29.270858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:29.271407 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:29.271767 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:29.272078 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 16:05:29.272321 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260924 16:05:29.273674 4838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:29.368961 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 4714
I20260924 16:05:29.385569 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 4849
Sep 24 16:05:29 dist-test-slave-1gc3 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.195@KRBTEST.COM
I20260924 16:05:29.405201 5023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:29.386676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54498 (local address 127.3.214.195:35945)
0924 16:05:29.387292 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:29.387298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:29.387318 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:05:29.387393 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:29.387397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:29.387488 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:29.387605 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:29.389103 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:29.389989 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:29.392015 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:29.392559 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:29.392697 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:29.397862 (+ 5165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:29.397889 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:29.397917 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:29.397977 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:29.401631 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:29.402757 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:29.402767 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:29.402770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:29.402968 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:29.403390 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:29.403393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:29.403395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:29.403894 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:29.404174 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:29.404505 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 16:05:29.404788 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":333,"threads_started":1}
I20260924 16:05:29.406333 4973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:29.508121 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 4849
I20260924 16:05:29.529896 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 4483
I20260924 16:05:29.531566 4546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:29.661662 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 4483
2026-09-24T16:05:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4629 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:05:29 dist-test-slave-1gc3 krb5kdc[5029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:05:29 dist-test-slave-1gc3 krb5kdc[5029](info): set up 2 sockets
Sep 24 16:05:29 dist-test-slave-1gc3 krb5kdc[5029](info): commencing operation
krb5kdc: starting...
W20260924 16:05:30.691993 3931 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:05:30 dist-test-slave-1gc3 krb5kdc[5029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:05:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:05:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:05:30.912449 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:39961
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:34861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:39961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:31.300908 5042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:31.301268 5042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:31.301384 5042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:31.309486 5042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:05:31.309749 5042 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 16:05:31.309818 5042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:31.309895 5042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:31.309955 5042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:05:31.310022 5042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:05:31.320947 5042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:34861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:39961
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:39961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:31.322448 5042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:31.323866 5042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:31.337561 5047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:31.337663 5048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:31.337607 5050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:31.338969 5042 server_base.cc:1034] running on GCE node
I20260924 16:05:31.339902 5042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:31.341593 5042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:31.342886 5042 hybrid_clock.cc:648] HybridClock initialized: now 1790265931342808 us; error 39 us; skew 500 ppm
I20260924 16:05:31.343272 5042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:31 dist-test-slave-1gc3 krb5kdc[5029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265931, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:31.346652 5042 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:05:31.348412 5042 webserver.cc:460] Webserver started at http://127.3.214.254:33991/ using document root <none> and password file <none>
I20260924 16:05:31.349165 5042 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:31.349258 5042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:31.349570 5042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:31.352977 5042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "0f00980684834b2cb732598908639eae"
format_stamp: "Formatted at 2026-09-24 16:05:31 on dist-test-slave-1gc3"
server_key: "387d4ec89e8152bc199a88b97a4442dd"
server_key_iv: "decbc2056c0323bb279090d6578028cd"
server_key_version: "encryptionkey@0"
I20260924 16:05:31.353853 5042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "0f00980684834b2cb732598908639eae"
format_stamp: "Formatted at 2026-09-24 16:05:31 on dist-test-slave-1gc3"
server_key: "387d4ec89e8152bc199a88b97a4442dd"
server_key_iv: "decbc2056c0323bb279090d6578028cd"
server_key_version: "encryptionkey@0"
I20260924 16:05:31.360275 5042 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260924 16:05:31.365451 5057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:31.367568 5042 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:05:31.367820 5042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "0f00980684834b2cb732598908639eae"
format_stamp: "Formatted at 2026-09-24 16:05:31 on dist-test-slave-1gc3"
server_key: "387d4ec89e8152bc199a88b97a4442dd"
server_key_iv: "decbc2056c0323bb279090d6578028cd"
server_key_version: "encryptionkey@0"
I20260924 16:05:31.368098 5042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:31.387422 5042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:31.392243 5042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:31.392490 5042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:31.416692 5042 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:39961
I20260924 16:05:31.416754 5119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:39961 every 8 connection(s)
I20260924 16:05:31.418356 5042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:05:31.423694 5120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:31.425554 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 5042
I20260924 16:05:31.425781 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:05:31.426568 3931 external_mini_cluster.cc:1428] Setting key 125764e2b4ab789633b0a293506e68f7
I20260924 16:05:31.442693 5120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae: Bootstrap starting.
Sep 24 16:05:31 dist-test-slave-1gc3 krb5kdc[5029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:31.447726 5120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:31.449723 5120 log.cc:824] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:31.455001 5120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae: No bootstrap required, opened a new log
I20260924 16:05:31.457113 5123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:31.429065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50386 (local address 127.3.214.254:39961)
0924 16:05:31.430230 (+ 1165us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:31.430254 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:31.430337 (+ 83us) server_negotiation.cc:407] Connection header received
0924 16:05:31.432685 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:31.432733 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:31.433576 (+ 843us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:31.434299 (+ 723us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:31.436006 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:31.438265 (+ 2259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:31.439766 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:31.440970 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:31.441196 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:31.445682 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:31.445698 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:31.445727 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:31.445805 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:31.450869 (+ 5064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:31.452106 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:31.452118 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:31.452131 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:31.452419 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:31.452944 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:31.452950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:31.452953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:31.453891 (+ 938us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:31.454310 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:31.454774 (+ 464us) server_negotiation.cc:299] Negotiation successful
0924 16:05:31.455399 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":420,"threads_started":1}
I20260924 16:05:31.465406 5120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f00980684834b2cb732598908639eae" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39961 } }
I20260924 16:05:31.465922 5120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:31.466009 5120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f00980684834b2cb732598908639eae, State: Initialized, Role: FOLLOWER
I20260924 16:05:31.466775 5120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f00980684834b2cb732598908639eae" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39961 } }
I20260924 16:05:31.467051 5120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:05:31.467177 5120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:05:31.467332 5120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:31.469965 5120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f00980684834b2cb732598908639eae" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39961 } }
I20260924 16:05:31.470768 5120 leader_election.cc:302] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f00980684834b2cb732598908639eae; no voters:
I20260924 16:05:31.471449 5120 leader_election.cc:288] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:05:31.471630 5125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:31.472479 5125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae [term 1 LEADER]: Becoming Leader. State: Replica: 0f00980684834b2cb732598908639eae, State: Running, Role: LEADER
I20260924 16:05:31.473250 5125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f00980684834b2cb732598908639eae" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39961 } }
I20260924 16:05:31.474187 5120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:05:31.477021 5127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f00980684834b2cb732598908639eae. Latest consensus state: current_term: 1 leader_uuid: "0f00980684834b2cb732598908639eae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f00980684834b2cb732598908639eae" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39961 } } }
I20260924 16:05:31.477545 5127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:31.476925 5126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f00980684834b2cb732598908639eae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f00980684834b2cb732598908639eae" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39961 } } }
I20260924 16:05:31.477775 5126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:31.483632 5132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:05:31.489691 5132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:05:31.502031 5132 catalog_manager.cc:1347] Generated new cluster ID: 8909aa5036f141ef909b7eac90559020
I20260924 16:05:31.502287 5132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:05:31.515928 5132 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:05:31.517637 5132 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:05:31.529981 5132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae: Generated new TSK 0
I20260924 16:05:31.531412 5132 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.214.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
I20260924 16:05:31.591221 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39961
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:34861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:32.002727 5146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:32.003166 5146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:32.003315 5146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:32.011655 5146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:32.011826 5146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:32.011958 5146 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:05:32.012080 5146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:05:32.023641 5146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:34861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.214.254:39961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:32.025600 5146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:32.027341 5146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:32.042989 5151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:32.043145 5152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:32.043133 5154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:32.044510 5146 server_base.cc:1034] running on GCE node
I20260924 16:05:32.045347 5146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:32.046567 5146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:32.047823 5146 hybrid_clock.cc:648] HybridClock initialized: now 1790265932047741 us; error 53 us; skew 500 ppm
I20260924 16:05:32.048187 5146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:32 dist-test-slave-1gc3 krb5kdc[5029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265932, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:32.056830 5146 init.cc:377] Logged in from keytab as kudu/127.3.214.193@KRBTEST.COM (short username kudu)
I20260924 16:05:32.058796 5146 webserver.cc:460] Webserver started at http://127.3.214.193:40145/ using document root <none> and password file <none>
I20260924 16:05:32.059643 5146 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:32.059886 5146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:32.060297 5146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:32.063799 5146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "8d152c4aaa6741cd9899e31b4137b550"
format_stamp: "Formatted at 2026-09-24 16:05:32 on dist-test-slave-1gc3"
server_key: "3cddcc249d1c03aa0695403d2423b072"
server_key_iv: "fb6026e5eb557cbca710ff3e1df6aa59"
server_key_version: "encryptionkey@0"
I20260924 16:05:32.064728 5146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "8d152c4aaa6741cd9899e31b4137b550"
format_stamp: "Formatted at 2026-09-24 16:05:32 on dist-test-slave-1gc3"
server_key: "3cddcc249d1c03aa0695403d2423b072"
server_key_iv: "fb6026e5eb557cbca710ff3e1df6aa59"
server_key_version: "encryptionkey@0"
I20260924 16:05:32.071488 5146 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 16:05:32.076413 5162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:32.078365 5146 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:05:32.078580 5146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "8d152c4aaa6741cd9899e31b4137b550"
format_stamp: "Formatted at 2026-09-24 16:05:32 on dist-test-slave-1gc3"
server_key: "3cddcc249d1c03aa0695403d2423b072"
server_key_iv: "fb6026e5eb557cbca710ff3e1df6aa59"
server_key_version: "encryptionkey@0"
I20260924 16:05:32.078827 5146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:32.102662 5146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:32.108765 5146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:32.109109 5146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:32.112542 5146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:32.112653 5146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:32.112805 5146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:32.112870 5146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:05:32 dist-test-slave-1gc3 krb5kdc[5029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265932, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:32.140890 5171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:32.113285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:40997 (local address 127.3.214.254:39961)
0924 16:05:32.114645 (+ 1360us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:32.114655 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:32.117265 (+ 2610us) server_negotiation.cc:407] Connection header received
0924 16:05:32.120073 (+ 2808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:32.120089 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:32.120205 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:32.120297 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:32.123322 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:32.124332 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:32.126980 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:32.127471 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:32.127618 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:32.132998 (+ 5380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:32.133006 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:32.133010 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:32.133062 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:32.136470 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:32.137557 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:32.137563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:32.137567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:32.137735 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:32.138855 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:32.138861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:32.138865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:32.139208 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:32.139357 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:32.140242 (+ 885us) server_negotiation.cc:299] Negotiation successful
0924 16:05:32.140498 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1163,"thread_start_us":997,"threads_started":1}
I20260924 16:05:32.142198 5172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:32.113935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39961 (local address 127.3.214.193:40997)
0924 16:05:32.116991 (+ 3056us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:32.117085 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:32.119622 (+ 2537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:32.121141 (+ 1519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:32.121157 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:32.121985 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:32.123114 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:32.123142 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:32.124493 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:32.124497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:32.126791 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:32.126811 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:32.127982 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:32.127990 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:32.128330 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:32.129256 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:32.129289 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:32.132704 (+ 3415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:32.136696 (+ 3992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:32.136704 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:32.136722 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:32.137407 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:32.137940 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:32.137952 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:32.137957 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:32.138635 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:32.139372 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:32.139383 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:32.139828 (+ 445us) client_negotiation.cc:769] Sending connection context
0924 16:05:32.140112 (+ 284us) client_negotiation.cc:240] Negotiation successful
0924 16:05:32.140733 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2680,"thread_start_us":2380,"threads_started":1}
I20260924 16:05:32.164142 5146 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:35793
I20260924 16:05:32.164160 5282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:35793 every 8 connection(s)
I20260924 16:05:32.165938 5146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:05:32.168500 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 5146
I20260924 16:05:32.168763 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:05:32.168794 5283 heartbeater.cc:347] Connected to a master server at 127.3.214.254:39961
I20260924 16:05:32.169034 5283 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:32.169631 3931 external_mini_cluster.cc:1428] Setting key 16f7e60eb73629802cbf6a170e099a58
I20260924 16:05:32.169751 5283 heartbeater.cc:511] Master 127.3.214.254:39961 requested a full tablet report, sending...
I20260924 16:05:32.172726 5073 ts_manager.cc:194] Registered new tserver with Master: 8d152c4aaa6741cd9899e31b4137b550 (127.3.214.193:35793)
I20260924 16:05:32.173105 5171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:32.150948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:34299 (local address 127.3.214.254:39961)
0924 16:05:32.151154 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:32.151161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:32.158932 (+ 7771us) server_negotiation.cc:407] Connection header received
0924 16:05:32.159156 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:32.159162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:32.159254 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:32.159351 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:32.161096 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:32.161953 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:32.163545 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:32.163962 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:32.164071 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:32.166165 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:32.166170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:32.166174 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:32.166205 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:32.169692 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:32.170654 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:32.170660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:32.170664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:32.170871 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:32.171410 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:32.171415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:32.171419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:32.171819 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:32.171965 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:32.172432 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 16:05:32.172709 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 16:05:32.173118 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:32.151041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39961 (local address 127.3.214.193:34299)
0924 16:05:32.158805 (+ 7764us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:32.158827 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:32.159009 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:32.159549 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:32.159554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:32.160321 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:32.160875 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:32.160891 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:32.162092 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:32.162121 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:32.163402 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:32.163418 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:32.164181 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:32.164189 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:32.164374 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:32.165109 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:32.165126 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:32.165913 (+ 787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:32.169899 (+ 3986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:32.169905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:32.169909 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:32.170525 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:32.171019 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:32.171026 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:32.171029 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:32.171245 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:32.171950 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:32.171955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:32.172158 (+ 203us) client_negotiation.cc:769] Sending connection context
0924 16:05:32.172285 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 16:05:32.172644 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7564,"thread_start_us":7352,"threads_started":1}
I20260924 16:05:32.175190 5073 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:40997
WARNING: no policy specified for kudu/127.3.214.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
I20260924 16:05:32.217959 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39961
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:34861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:32.588351 5287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:32.588671 5287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:32.588773 5287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:32.599241 5287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:32.599412 5287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:32.599561 5287 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:05:32.599695 5287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:05:32.610635 5287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:34861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.214.254:39961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:32.612075 5287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:32.613480 5287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:32.628265 5293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:32.628449 5295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:32.628280 5292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:32.631778 5287 server_base.cc:1034] running on GCE node
I20260924 16:05:32.632601 5287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:32.634027 5287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:32.635344 5287 hybrid_clock.cc:648] HybridClock initialized: now 1790265932635240 us; error 68 us; skew 500 ppm
I20260924 16:05:32.635759 5287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:32 dist-test-slave-1gc3 krb5kdc[5029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265932, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:32.639806 5287 init.cc:377] Logged in from keytab as kudu/127.3.214.194@KRBTEST.COM (short username kudu)
I20260924 16:05:32.641633 5287 webserver.cc:460] Webserver started at http://127.3.214.194:38661/ using document root <none> and password file <none>
I20260924 16:05:32.642597 5287 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:32.642722 5287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:32.643102 5287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:32.646780 5287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "a7ccd1b3730a483cb15938d6acc17936"
format_stamp: "Formatted at 2026-09-24 16:05:32 on dist-test-slave-1gc3"
server_key: "84a172127e4ebc7e5c2a0c775d27cd29"
server_key_iv: "65987eb0d5960496e90182bd6dde32ba"
server_key_version: "encryptionkey@0"
I20260924 16:05:32.647703 5287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "a7ccd1b3730a483cb15938d6acc17936"
format_stamp: "Formatted at 2026-09-24 16:05:32 on dist-test-slave-1gc3"
server_key: "84a172127e4ebc7e5c2a0c775d27cd29"
server_key_iv: "65987eb0d5960496e90182bd6dde32ba"
server_key_version: "encryptionkey@0"
I20260924 16:05:32.654181 5287 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 16:05:32.658589 5302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:32.660337 5287 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260924 16:05:32.660535 5287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "a7ccd1b3730a483cb15938d6acc17936"
format_stamp: "Formatted at 2026-09-24 16:05:32 on dist-test-slave-1gc3"
server_key: "84a172127e4ebc7e5c2a0c775d27cd29"
server_key_iv: "65987eb0d5960496e90182bd6dde32ba"
server_key_version: "encryptionkey@0"
I20260924 16:05:32.660864 5287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:32.704164 5287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:32.709609 5287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:32.710067 5287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:32.713871 5287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:32.714002 5287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:32.714188 5287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:32.714258 5287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 24 16:05:32 dist-test-slave-1gc3 krb5kdc[5029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265932, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:32.743362 5311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:32.715530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:55603 (local address 127.3.214.254:39961)
0924 16:05:32.716155 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:32.716163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:32.717474 (+ 1311us) server_negotiation.cc:407] Connection header received
0924 16:05:32.720155 (+ 2681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:32.720162 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:32.720282 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:32.720368 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:32.723351 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:32.724356 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:32.727026 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:32.727621 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:32.727755 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:32.734182 (+ 6427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:32.734191 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:32.734197 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:32.734268 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:32.738317 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:32.739848 (+ 1531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:32.739856 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:32.739859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:32.740064 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:32.740821 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:32.740827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:32.740831 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:32.741282 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:32.741459 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:32.742599 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0924 16:05:32.742908 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":289,"threads_started":1}
I20260924 16:05:32.745603 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:32.716160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39961 (local address 127.3.214.194:55603)
0924 16:05:32.717240 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:32.717298 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:32.719727 (+ 2429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:32.721070 (+ 1343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:32.721080 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:32.722040 (+ 960us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:32.723102 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:32.723128 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:32.724527 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:32.724537 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:32.726794 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:32.726812 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:32.728169 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:32.728187 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:32.728653 (+ 466us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:32.729789 (+ 1136us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:32.729820 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:32.733620 (+ 3800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:32.738604 (+ 4984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:32.738620 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:32.738654 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:32.739616 (+ 962us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:32.740239 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:32.740245 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:32.740250 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:32.740649 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:32.741454 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:32.741469 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:32.741997 (+ 528us) client_negotiation.cc:769] Sending connection context
0924 16:05:32.742466 (+ 469us) client_negotiation.cc:240] Negotiation successful
0924 16:05:32.743284 (+ 818us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":430,"threads_started":1}
I20260924 16:05:32.764320 5423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:43303 every 8 connection(s)
I20260924 16:05:32.764315 5287 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:43303
I20260924 16:05:32.766073 5287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:05:32.769410 5424 heartbeater.cc:347] Connected to a master server at 127.3.214.254:39961
I20260924 16:05:32.769688 5424 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:32.770565 5424 heartbeater.cc:511] Master 127.3.214.254:39961 requested a full tablet report, sending...
I20260924 16:05:32.771807 5311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:32.754621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:45723 (local address 127.3.214.254:39961)
0924 16:05:32.754892 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:32.754900 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:32.755639 (+ 739us) server_negotiation.cc:407] Connection header received
0924 16:05:32.756045 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:32.756053 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:32.756180 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:32.756301 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:32.757764 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:32.758991 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:32.760998 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:32.761749 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:32.761888 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:32.765096 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:32.765101 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:32.765105 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:32.765145 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:32.768478 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:32.769382 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:32.769387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:32.769389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:32.769538 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:32.770028 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:32.770034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:32.770037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:32.770474 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:32.770629 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:32.771151 (+ 522us) server_negotiation.cc:299] Negotiation successful
0924 16:05:32.771409 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:05:32.771962 5409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:32.754623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39961 (local address 127.3.214.194:45723)
0924 16:05:32.755486 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:32.755512 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:32.755803 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:32.756416 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:32.756420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:32.757010 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:32.757571 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:32.757586 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:32.759208 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:32.759217 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:32.760802 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:32.760819 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:32.762253 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:32.762258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:32.762410 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:32.763234 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:32.763272 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:32.764829 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:32.768634 (+ 3805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:32.768639 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:32.768642 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:32.769248 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:32.769683 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:32.769689 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:32.769693 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:32.769904 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:32.770604 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:32.770608 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:32.770886 (+ 278us) client_negotiation.cc:769] Sending connection context
0924 16:05:32.771025 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 16:05:32.771484 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":486,"threads_started":1}
I20260924 16:05:32.772687 5073 ts_manager.cc:194] Registered new tserver with Master: a7ccd1b3730a483cb15938d6acc17936 (127.3.214.194:43303)
I20260924 16:05:32.773734 5073 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:55603
I20260924 16:05:32.776407 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 5287
I20260924 16:05:32.776657 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:05:32.777195 3931 external_mini_cluster.cc:1428] Setting key ae8b5838546496547600265d770de703
WARNING: no policy specified for kudu/127.3.214.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
I20260924 16:05:32.822494 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39961
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:34861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:05:33.179498 5283 heartbeater.cc:503] Master 127.3.214.254:39961 was elected leader, sending a full tablet report...
W20260924 16:05:33.194631 5428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:33.195057 5428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:33.195173 5428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:33.203811 5428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:33.203980 5428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:33.204083 5428 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:05:33.204208 5428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:05:33.216190 5428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:34861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.214.254:39961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:33.218130 5428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:33.219897 5428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:33.236763 5433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:33.236838 5434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:33.236932 5436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:33.238471 5428 server_base.cc:1034] running on GCE node
I20260924 16:05:33.239431 5428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:33.240906 5428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:33.242257 5428 hybrid_clock.cc:648] HybridClock initialized: now 1790265933242153 us; error 53 us; skew 500 ppm
I20260924 16:05:33.242715 5428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:33 dist-test-slave-1gc3 krb5kdc[5029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265933, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:33.247018 5428 init.cc:377] Logged in from keytab as kudu/127.3.214.195@KRBTEST.COM (short username kudu)
I20260924 16:05:33.249053 5428 webserver.cc:460] Webserver started at http://127.3.214.195:38401/ using document root <none> and password file <none>
I20260924 16:05:33.250051 5428 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:33.250226 5428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:33.250635 5428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:33.254725 5428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "3f1dd2529dca4aa1a50ca465d03824fd"
format_stamp: "Formatted at 2026-09-24 16:05:33 on dist-test-slave-1gc3"
server_key: "26158f994c8b4e00608cd852c60f7963"
server_key_iv: "6c486803b7c8fec26dbf9804214eda1d"
server_key_version: "encryptionkey@0"
I20260924 16:05:33.255805 5428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "3f1dd2529dca4aa1a50ca465d03824fd"
format_stamp: "Formatted at 2026-09-24 16:05:33 on dist-test-slave-1gc3"
server_key: "26158f994c8b4e00608cd852c60f7963"
server_key_iv: "6c486803b7c8fec26dbf9804214eda1d"
server_key_version: "encryptionkey@0"
I20260924 16:05:33.263087 5428 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260924 16:05:33.268464 5444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:33.270539 5428 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:05:33.270840 5428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "3f1dd2529dca4aa1a50ca465d03824fd"
format_stamp: "Formatted at 2026-09-24 16:05:33 on dist-test-slave-1gc3"
server_key: "26158f994c8b4e00608cd852c60f7963"
server_key_iv: "6c486803b7c8fec26dbf9804214eda1d"
server_key_version: "encryptionkey@0"
I20260924 16:05:33.271145 5428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:33.286512 5428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:33.292220 5428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:33.292564 5428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:33.296177 5428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:33.296299 5428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:33.296454 5428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:33.296518 5428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:05:33 dist-test-slave-1gc3 krb5kdc[5029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265933, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:33.327584 5453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:33.296901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:33249 (local address 127.3.214.254:39961)
0924 16:05:33.297655 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:33.297663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:33.299166 (+ 1503us) server_negotiation.cc:407] Connection header received
0924 16:05:33.302956 (+ 3790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:33.302963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:33.303078 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:33.303209 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:33.306637 (+ 3428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.307430 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.310587 (+ 3157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.311121 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.311299 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:33.317804 (+ 6505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:33.317810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:33.317814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:33.317856 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:33.321409 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:33.323149 (+ 1740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:33.323157 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:33.323160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:33.323353 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:33.324230 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:33.324235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:33.324238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:33.324672 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:33.324843 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:33.326536 (+ 1693us) server_negotiation.cc:299] Negotiation successful
0924 16:05:33.326978 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":393,"threads_started":1}
I20260924 16:05:33.328006 5454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:33.297638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39961 (local address 127.3.214.195:33249)
0924 16:05:33.298860 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:33.298936 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:33.302410 (+ 3474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:33.303639 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:33.303659 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:33.304872 (+ 1213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:33.306277 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:33.306319 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.307646 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.307653 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:33.310299 (+ 2646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:33.310325 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.311549 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.311558 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:33.312017 (+ 459us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:33.313306 (+ 1289us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:33.313348 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:33.317461 (+ 4113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:33.321679 (+ 4218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:33.321702 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:33.321732 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:33.322889 (+ 1157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:33.323529 (+ 640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:33.323536 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:33.323541 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:33.324041 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:33.324959 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:33.324971 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:33.325654 (+ 683us) client_negotiation.cc:769] Sending connection context
0924 16:05:33.326071 (+ 417us) client_negotiation.cc:240] Negotiation successful
0924 16:05:33.326705 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":486,"threads_started":1}
I20260924 16:05:33.363313 5428 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:40453
I20260924 16:05:33.363313 5566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:40453 every 8 connection(s)
I20260924 16:05:33.364789 5453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:33.341864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:42599 (local address 127.3.214.254:39961)
0924 16:05:33.342170 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:33.342177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:33.347534 (+ 5357us) server_negotiation.cc:407] Connection header received
0924 16:05:33.347711 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:33.347720 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:33.347898 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:33.348108 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:33.349283 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.350650 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.352257 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.353099 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.353310 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:33.356584 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:33.356593 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:33.356598 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:33.356657 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:33.360816 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:33.362455 (+ 1639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:33.362465 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:33.362469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:33.362687 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:33.363149 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:33.363154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:33.363156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:33.363486 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:33.363662 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:33.364082 (+ 420us) server_negotiation.cc:299] Negotiation successful
0924 16:05:33.364360 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260924 16:05:33.364853 5531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:33.341861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39961 (local address 127.3.214.195:42599)
0924 16:05:33.347331 (+ 5470us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:33.347355 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:33.347529 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:33.348126 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:33.348129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:33.348667 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:33.349081 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:33.349096 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.350872 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.350876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:33.352057 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:33.352075 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.353377 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.353383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:33.353532 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:33.354391 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:33.354425 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:33.356214 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:33.361039 (+ 4825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:33.361044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:33.361048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:33.361635 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:33.362848 (+ 1213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:33.362852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:33.362854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:33.363008 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:33.363617 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:33.363622 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:33.363818 (+ 196us) client_negotiation.cc:769] Sending connection context
0924 16:05:33.363944 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 16:05:33.364365 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5294,"thread_start_us":5113,"threads_started":1}
I20260924 16:05:33.365361 5428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:05:33.367133 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 5428
I20260924 16:05:33.367337 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:05:33.367941 3931 external_mini_cluster.cc:1428] Setting key 0c3fa5b366a1642a4aa6f278ec255349
I20260924 16:05:33.368739 5567 heartbeater.cc:347] Connected to a master server at 127.3.214.254:39961
I20260924 16:05:33.369127 5567 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:33.371171 5567 heartbeater.cc:511] Master 127.3.214.254:39961 requested a full tablet report, sending...
I20260924 16:05:33.374146 5073 ts_manager.cc:194] Registered new tserver with Master: 3f1dd2529dca4aa1a50ca465d03824fd (127.3.214.195:40453)
I20260924 16:05:33.375402 5073 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:33249
I20260924 16:05:33.385710 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 16:05:33 dist-test-slave-1gc3 krb5kdc[5029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265933, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 16:05:33 dist-test-slave-1gc3 krb5kdc[5029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265933, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:33.447595 5453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:33.429755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50400 (local address 127.3.214.254:39961)
0924 16:05:33.430051 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:33.430059 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:33.430362 (+ 303us) server_negotiation.cc:407] Connection header received
0924 16:05:33.430622 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:33.430628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:33.430725 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:33.430820 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:33.432606 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.433437 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.434881 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.435354 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.435507 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:33.440621 (+ 5114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:33.440627 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:33.440632 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:33.440687 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:33.443785 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:33.444976 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:33.444982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:33.444985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:33.445211 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:33.445808 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:33.445813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:33.445816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:33.446273 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:33.446564 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:33.447004 (+ 440us) server_negotiation.cc:299] Negotiation successful
0924 16:05:33.447264 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260924 16:05:33.453369 5073 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:05:33.461751 5073 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:05:33.503917 5580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:33.479330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:40453 (local address 127.0.0.1:48140)
0924 16:05:33.485046 (+ 5716us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:33.485116 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:33.485559 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:33.489229 (+ 3670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:33.489242 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:33.489293 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:33.489981 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:33.489998 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.494810 (+ 4812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.494818 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:33.500178 (+ 5360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:33.500209 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.502731 (+ 2522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.502738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:33.503018 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:33.503077 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 16:05:33.503217 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 16:05:33.503480 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5269,"thread_start_us":5060,"threads_started":1}
I20260924 16:05:33.503917 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:33.479767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48140 (local address 127.3.214.195:40453)
0924 16:05:33.480574 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:33.480598 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:33.485306 (+ 4708us) server_negotiation.cc:407] Connection header received
0924 16:05:33.485927 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:33.485940 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:33.486433 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:33.486716 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:33.490349 (+ 3633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.494520 (+ 4171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.500498 (+ 5978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.502434 (+ 1936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.502585 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:33.502901 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:33.503333 (+ 432us) server_negotiation.cc:299] Negotiation successful
0924 16:05:33.503534 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":754,"threads_started":1}
I20260924 16:05:33.509073 5495 tablet_service.cc:1467] Processing CreateTablet for tablet da2cae344def4201877c4426c949dd0c (DEFAULT_TABLE table=test-table [id=d89df375b14340adbd6aa4b0bddb0a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:33.510293 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:33.479502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:43303 (local address 127.0.0.1:59670)
0924 16:05:33.499987 (+ 20485us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:33.500013 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:33.500226 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:33.502206 (+ 1980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:33.502210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:33.502239 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:33.502707 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:33.502720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.504920 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.504924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:33.507226 (+ 2302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:33.507244 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.509181 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.509189 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:33.509389 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:33.509425 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 16:05:33.509537 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 16:05:33.509736 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20290,"mutex_wait_us":31,"thread_start_us":21344,"threads_started":1}
I20260924 16:05:33.510715 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:33.480047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59670 (local address 127.3.214.194:43303)
0924 16:05:33.494721 (+ 14674us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:33.494735 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:33.501306 (+ 6571us) server_negotiation.cc:407] Connection header received
0924 16:05:33.501533 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:33.501544 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:33.501897 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:33.502197 (+ 300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:33.502961 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.504740 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.507452 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.508876 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.509089 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:33.509340 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:33.509643 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 16:05:33.509841 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13985,"thread_start_us":13826,"threads_started":1}
I20260924 16:05:33.511052 5584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:33.480105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:35793 (local address 127.0.0.1:57908)
0924 16:05:33.500814 (+ 20709us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:33.500834 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:33.500984 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:33.501864 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:33.501869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:33.501896 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:33.502471 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:33.502485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.505498 (+ 3013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.505504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:33.507610 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:33.507627 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.509665 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.509671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:33.509853 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:33.509887 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 16:05:33.509967 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 16:05:33.510144 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20571,"mutex_wait_us":1,"thread_start_us":20428,"threads_started":1}
I20260924 16:05:33.511281 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:33.480386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57908 (local address 127.3.214.193:35793)
0924 16:05:33.497018 (+ 16632us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:33.497024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:33.501147 (+ 4123us) server_negotiation.cc:407] Connection header received
0924 16:05:33.501315 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:33.501326 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:33.501652 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:33.501870 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:33.502723 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.504731 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.507847 (+ 3116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.509054 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.509217 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:33.509486 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:33.510497 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0924 16:05:33.510741 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16486,"thread_start_us":16342,"threads_started":1}
I20260924 16:05:33.511389 5495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da2cae344def4201877c4426c949dd0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:33.515664 5353 tablet_service.cc:1467] Processing CreateTablet for tablet da2cae344def4201877c4426c949dd0c (DEFAULT_TABLE table=test-table [id=d89df375b14340adbd6aa4b0bddb0a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:33.516079 5213 tablet_service.cc:1467] Processing CreateTablet for tablet da2cae344def4201877c4426c949dd0c (DEFAULT_TABLE table=test-table [id=d89df375b14340adbd6aa4b0bddb0a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:33.517740 5353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da2cae344def4201877c4426c949dd0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:33.517963 5213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da2cae344def4201877c4426c949dd0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:33.531102 5586 tablet_bootstrap.cc:492] T da2cae344def4201877c4426c949dd0c P 3f1dd2529dca4aa1a50ca465d03824fd: Bootstrap starting.
I20260924 16:05:33.533790 5587 tablet_bootstrap.cc:492] T da2cae344def4201877c4426c949dd0c P a7ccd1b3730a483cb15938d6acc17936: Bootstrap starting.
I20260924 16:05:33.536067 5586 tablet_bootstrap.cc:654] T da2cae344def4201877c4426c949dd0c P 3f1dd2529dca4aa1a50ca465d03824fd: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:33.537791 5588 tablet_bootstrap.cc:492] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550: Bootstrap starting.
I20260924 16:05:33.538044 5586 log.cc:824] T da2cae344def4201877c4426c949dd0c P 3f1dd2529dca4aa1a50ca465d03824fd: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:33.539274 5587 tablet_bootstrap.cc:654] T da2cae344def4201877c4426c949dd0c P a7ccd1b3730a483cb15938d6acc17936: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:33.540505 5587 log.cc:824] T da2cae344def4201877c4426c949dd0c P a7ccd1b3730a483cb15938d6acc17936: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:33.542830 5588 tablet_bootstrap.cc:654] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:33.543843 5587 tablet_bootstrap.cc:492] T da2cae344def4201877c4426c949dd0c P a7ccd1b3730a483cb15938d6acc17936: No bootstrap required, opened a new log
I20260924 16:05:33.544152 5587 ts_tablet_manager.cc:1397] T da2cae344def4201877c4426c949dd0c P a7ccd1b3730a483cb15938d6acc17936: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260924 16:05:33.544319 5588 log.cc:824] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:33.548020 5586 tablet_bootstrap.cc:492] T da2cae344def4201877c4426c949dd0c P 3f1dd2529dca4aa1a50ca465d03824fd: No bootstrap required, opened a new log
I20260924 16:05:33.548007 5588 tablet_bootstrap.cc:492] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550: No bootstrap required, opened a new log
I20260924 16:05:33.548586 5588 ts_tablet_manager.cc:1397] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.002s
I20260924 16:05:33.548586 5586 ts_tablet_manager.cc:1397] T da2cae344def4201877c4426c949dd0c P 3f1dd2529dca4aa1a50ca465d03824fd: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.003s
I20260924 16:05:33.552898 5587 raft_consensus.cc:348] T da2cae344def4201877c4426c949dd0c P a7ccd1b3730a483cb15938d6acc17936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } }
I20260924 16:05:33.553330 5587 raft_consensus.cc:374] T da2cae344def4201877c4426c949dd0c P a7ccd1b3730a483cb15938d6acc17936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:33.553422 5587 raft_consensus.cc:729] T da2cae344def4201877c4426c949dd0c P a7ccd1b3730a483cb15938d6acc17936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ccd1b3730a483cb15938d6acc17936, State: Initialized, Role: FOLLOWER
I20260924 16:05:33.554071 5587 consensus_queue.cc:260] T da2cae344def4201877c4426c949dd0c P a7ccd1b3730a483cb15938d6acc17936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } }
I20260924 16:05:33.555474 5424 heartbeater.cc:503] Master 127.3.214.254:39961 was elected leader, sending a full tablet report...
I20260924 16:05:33.555994 5587 ts_tablet_manager.cc:1428] T da2cae344def4201877c4426c949dd0c P a7ccd1b3730a483cb15938d6acc17936: Time spent starting tablet: real 0.012s user 0.001s sys 0.012s
I20260924 16:05:33.560103 5586 raft_consensus.cc:348] T da2cae344def4201877c4426c949dd0c P 3f1dd2529dca4aa1a50ca465d03824fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } }
I20260924 16:05:33.560613 5586 raft_consensus.cc:374] T da2cae344def4201877c4426c949dd0c P 3f1dd2529dca4aa1a50ca465d03824fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:33.560732 5586 raft_consensus.cc:729] T da2cae344def4201877c4426c949dd0c P 3f1dd2529dca4aa1a50ca465d03824fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1dd2529dca4aa1a50ca465d03824fd, State: Initialized, Role: FOLLOWER
I20260924 16:05:33.560664 5588 raft_consensus.cc:348] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } }
I20260924 16:05:33.561182 5588 raft_consensus.cc:374] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:33.561288 5588 raft_consensus.cc:729] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d152c4aaa6741cd9899e31b4137b550, State: Initialized, Role: FOLLOWER
I20260924 16:05:33.561556 5586 consensus_queue.cc:260] T da2cae344def4201877c4426c949dd0c P 3f1dd2529dca4aa1a50ca465d03824fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } }
I20260924 16:05:33.561987 5588 consensus_queue.cc:260] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } }
I20260924 16:05:33.563293 5567 heartbeater.cc:503] Master 127.3.214.254:39961 was elected leader, sending a full tablet report...
I20260924 16:05:33.564095 5586 ts_tablet_manager.cc:1428] T da2cae344def4201877c4426c949dd0c P 3f1dd2529dca4aa1a50ca465d03824fd: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260924 16:05:33.566876 5588 ts_tablet_manager.cc:1428] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550: Time spent starting tablet: real 0.018s user 0.012s sys 0.005s
W20260924 16:05:33.623654 5568 tablet.cc:2369] T da2cae344def4201877c4426c949dd0c P 3f1dd2529dca4aa1a50ca465d03824fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:05:33.634648 5594 raft_consensus.cc:482] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:05:33.635074 5594 raft_consensus.cc:504] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } }
I20260924 16:05:33.637233 5594 leader_election.cc:288] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f1dd2529dca4aa1a50ca465d03824fd (127.3.214.195:40453), a7ccd1b3730a483cb15938d6acc17936 (127.3.214.194:43303)
I20260924 16:05:33.646827 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:33.637824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:40041 (local address 127.3.214.194:43303)
0924 16:05:33.638072 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:33.638079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:33.639147 (+ 1068us) server_negotiation.cc:407] Connection header received
0924 16:05:33.639396 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:33.639402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:33.639494 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:33.639649 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:33.640403 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.641155 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.643851 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.645257 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.645478 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:33.645562 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:33.646318 (+ 756us) server_negotiation.cc:299] Negotiation successful
0924 16:05:33.646499 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 16:05:33.646828 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:33.637780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:59715 (local address 127.3.214.195:40453)
0924 16:05:33.638011 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:33.638020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:33.639017 (+ 997us) server_negotiation.cc:407] Connection header received
0924 16:05:33.639251 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:33.639258 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:33.639352 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:33.639431 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:33.640221 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.641258 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.643755 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.645295 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.645478 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:33.645562 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:33.646396 (+ 834us) server_negotiation.cc:299] Negotiation successful
0924 16:05:33.646551 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 16:05:33.647169 5596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:33.637848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:43303 (local address 127.3.214.193:40041)
0924 16:05:33.638966 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:33.638988 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:33.639193 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:33.639674 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:33.639679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:33.639722 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:33.640240 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:33.640254 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.641299 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.641304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:33.643561 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:33.643578 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.645702 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.645709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:33.645974 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:33.646050 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 16:05:33.646253 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 16:05:33.646551 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"mutex_wait_us":187,"thread_start_us":598,"threads_started":1}
I20260924 16:05:33.647200 5595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:33.637815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:40453 (local address 127.3.214.193:59715)
0924 16:05:33.638863 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:33.638882 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:33.639048 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:33.639502 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:33.639507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:33.639536 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:33.640044 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:33.640060 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.641415 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.641419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:33.643542 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:33.643561 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.645633 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.645642 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:33.645978 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:33.646057 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 16:05:33.646236 (+ 179us) client_negotiation.cc:240] Negotiation successful
0924 16:05:33.646551 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"mutex_wait_us":154,"thread_start_us":536,"threads_started":1}
I20260924 16:05:33.648577 5373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2cae344def4201877c4426c949dd0c" candidate_uuid: "8d152c4aaa6741cd9899e31b4137b550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7ccd1b3730a483cb15938d6acc17936" is_pre_election: true
I20260924 16:05:33.648700 5516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2cae344def4201877c4426c949dd0c" candidate_uuid: "8d152c4aaa6741cd9899e31b4137b550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" is_pre_election: true
I20260924 16:05:33.649221 5373 raft_consensus.cc:2454] T da2cae344def4201877c4426c949dd0c P a7ccd1b3730a483cb15938d6acc17936 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d152c4aaa6741cd9899e31b4137b550 in term 0.
I20260924 16:05:33.649401 5516 raft_consensus.cc:2454] T da2cae344def4201877c4426c949dd0c P 3f1dd2529dca4aa1a50ca465d03824fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d152c4aaa6741cd9899e31b4137b550 in term 0.
I20260924 16:05:33.650174 5164 leader_election.cc:302] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d152c4aaa6741cd9899e31b4137b550, a7ccd1b3730a483cb15938d6acc17936; no voters:
I20260924 16:05:33.650795 5594 raft_consensus.cc:2781] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:05:33.650981 5594 raft_consensus.cc:482] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:05:33.651081 5594 raft_consensus.cc:3037] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:33.653452 5594 raft_consensus.cc:504] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } }
I20260924 16:05:33.654217 5594 leader_election.cc:288] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550 [CANDIDATE]: Term 1 election: Requested vote from peers 3f1dd2529dca4aa1a50ca465d03824fd (127.3.214.195:40453), a7ccd1b3730a483cb15938d6acc17936 (127.3.214.194:43303)
I20260924 16:05:33.655092 5373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2cae344def4201877c4426c949dd0c" candidate_uuid: "8d152c4aaa6741cd9899e31b4137b550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7ccd1b3730a483cb15938d6acc17936"
I20260924 16:05:33.655088 5516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2cae344def4201877c4426c949dd0c" candidate_uuid: "8d152c4aaa6741cd9899e31b4137b550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1dd2529dca4aa1a50ca465d03824fd"
I20260924 16:05:33.655409 5516 raft_consensus.cc:3037] T da2cae344def4201877c4426c949dd0c P 3f1dd2529dca4aa1a50ca465d03824fd [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:33.655409 5373 raft_consensus.cc:3037] T da2cae344def4201877c4426c949dd0c P a7ccd1b3730a483cb15938d6acc17936 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:33.659248 5373 raft_consensus.cc:2454] T da2cae344def4201877c4426c949dd0c P a7ccd1b3730a483cb15938d6acc17936 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d152c4aaa6741cd9899e31b4137b550 in term 1.
I20260924 16:05:33.659269 5516 raft_consensus.cc:2454] T da2cae344def4201877c4426c949dd0c P 3f1dd2529dca4aa1a50ca465d03824fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d152c4aaa6741cd9899e31b4137b550 in term 1.
I20260924 16:05:33.660035 5164 leader_election.cc:302] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3f1dd2529dca4aa1a50ca465d03824fd, 8d152c4aaa6741cd9899e31b4137b550, a7ccd1b3730a483cb15938d6acc17936; no voters:
I20260924 16:05:33.660558 5594 raft_consensus.cc:2781] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:33.661151 5594 raft_consensus.cc:686] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550 [term 1 LEADER]: Becoming Leader. State: Replica: 8d152c4aaa6741cd9899e31b4137b550, State: Running, Role: LEADER
I20260924 16:05:33.661769 5594 consensus_queue.cc:237] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } }
I20260924 16:05:33.667043 5074 catalog_manager.cc:5495] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d152c4aaa6741cd9899e31b4137b550 (127.3.214.193). New cstate: current_term: 1 leader_uuid: "8d152c4aaa6741cd9899e31b4137b550" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } health_report { overall_health: HEALTHY } } }
W20260924 16:05:33.669739 5284 tablet.cc:2369] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:05:33.705199 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:33.699246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57918 (local address 127.3.214.193:35793)
0924 16:05:33.699548 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:33.699555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:33.699579 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:05:33.699957 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:33.699961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:33.700073 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:33.700203 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:33.700635 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.701353 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.702820 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.703228 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.703341 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:33.703890 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:33.704526 (+ 636us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:33.704665 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:33.704806 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 16:05:33.704904 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260924 16:05:33.717144 5516 raft_consensus.cc:1261] T da2cae344def4201877c4426c949dd0c P 3f1dd2529dca4aa1a50ca465d03824fd [term 1 FOLLOWER]: Refusing update from remote peer 8d152c4aaa6741cd9899e31b4137b550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:33.717296 5373 raft_consensus.cc:1261] T da2cae344def4201877c4426c949dd0c P a7ccd1b3730a483cb15938d6acc17936 [term 1 FOLLOWER]: Refusing update from remote peer 8d152c4aaa6741cd9899e31b4137b550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:33.718568 5597 consensus_queue.cc:1035] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:33.719027 5594 consensus_queue.cc:1035] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:33.734285 5601 mvcc.cc:205] Tried to move back new op lower bound from 7332929264490139648 to 7332929264282750976. Current Snapshot: MvccSnapshot[applied={T|T < 7332929264490139648}]
I20260924 16:05:33.736182 5603 mvcc.cc:205] Tried to move back new op lower bound from 7332929264490139648 to 7332929264282750976. Current Snapshot: MvccSnapshot[applied={T|T < 7332929264490139648}]
I20260924 16:05:33.760969 5073 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:05:33.761422 5073 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:05:33.767246 5073 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae: Sending DeleteTablet for 3 replicas of tablet da2cae344def4201877c4426c949dd0c
I20260924 16:05:33.769090 5495 tablet_service.cc:1514] Processing DeleteTablet for tablet da2cae344def4201877c4426c949dd0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:33 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:48140
I20260924 16:05:33.769906 5213 tablet_service.cc:1514] Processing DeleteTablet for tablet da2cae344def4201877c4426c949dd0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:33 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:57908
I20260924 16:05:33.770526 5073 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 16:05:33.771095 5353 tablet_service.cc:1514] Processing DeleteTablet for tablet da2cae344def4201877c4426c949dd0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:33 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:59670
W20260924 16:05:33.771481 5073 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:05:33.775321 5612 tablet_replica.cc:330] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550: stopping tablet replica
I20260924 16:05:33.776230 5612 raft_consensus.cc:2229] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:05:33.777299 5612 raft_consensus.cc:2258] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:33.780148 5612 ts_tablet_manager.cc:1905] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:33.787361 5494 tablet_service.cc:1467] Processing CreateTablet for tablet af72740ecfc6424ab3ae6eb629f6b77f (DEFAULT_TABLE table=test-table [id=f6a0c50669584a62a6da304af022a73a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:33.788012 5494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af72740ecfc6424ab3ae6eb629f6b77f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:33.791302 5352 tablet_service.cc:1467] Processing CreateTablet for tablet af72740ecfc6424ab3ae6eb629f6b77f (DEFAULT_TABLE table=test-table [id=f6a0c50669584a62a6da304af022a73a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:33.791342 5213 tablet_service.cc:1467] Processing CreateTablet for tablet af72740ecfc6424ab3ae6eb629f6b77f (DEFAULT_TABLE table=test-table [id=f6a0c50669584a62a6da304af022a73a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:33.791826 5213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af72740ecfc6424ab3ae6eb629f6b77f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:33.791958 5352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af72740ecfc6424ab3ae6eb629f6b77f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:33.800647 5612 ts_tablet_manager.cc:1918] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:05:33.800987 5612 log.cc:1196] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-0/wal/wals/da2cae344def4201877c4426c949dd0c
I20260924 16:05:33.801669 5612 ts_tablet_manager.cc:1939] T da2cae344def4201877c4426c949dd0c P 8d152c4aaa6741cd9899e31b4137b550: Deleting consensus metadata
I20260924 16:05:33.802518 5587 tablet_bootstrap.cc:492] T af72740ecfc6424ab3ae6eb629f6b77f P a7ccd1b3730a483cb15938d6acc17936: Bootstrap starting.
I20260924 16:05:33.804569 5587 tablet_bootstrap.cc:654] T af72740ecfc6424ab3ae6eb629f6b77f P a7ccd1b3730a483cb15938d6acc17936: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:33.806341 5586 tablet_bootstrap.cc:492] T af72740ecfc6424ab3ae6eb629f6b77f P 3f1dd2529dca4aa1a50ca465d03824fd: Bootstrap starting.
I20260924 16:05:33.808615 5586 tablet_bootstrap.cc:654] T af72740ecfc6424ab3ae6eb629f6b77f P 3f1dd2529dca4aa1a50ca465d03824fd: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:33.804384 5060 catalog_manager.cc:4841] TS 8d152c4aaa6741cd9899e31b4137b550 (127.3.214.193:35793): tablet da2cae344def4201877c4426c949dd0c (table test-table [id=d89df375b14340adbd6aa4b0bddb0a65]) successfully deleted
I20260924 16:05:33.815112 5588 tablet_bootstrap.cc:492] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550: Bootstrap starting.
I20260924 16:05:33.816205 5613 tablet_replica.cc:330] T da2cae344def4201877c4426c949dd0c P a7ccd1b3730a483cb15938d6acc17936: stopping tablet replica
I20260924 16:05:33.816890 5613 raft_consensus.cc:2229] T da2cae344def4201877c4426c949dd0c P a7ccd1b3730a483cb15938d6acc17936 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:05:33.817258 5588 tablet_bootstrap.cc:654] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:33.817572 5613 raft_consensus.cc:2258] T da2cae344def4201877c4426c949dd0c P a7ccd1b3730a483cb15938d6acc17936 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:33.818037 5586 tablet_bootstrap.cc:492] T af72740ecfc6424ab3ae6eb629f6b77f P 3f1dd2529dca4aa1a50ca465d03824fd: No bootstrap required, opened a new log
I20260924 16:05:33.818280 5586 ts_tablet_manager.cc:1397] T af72740ecfc6424ab3ae6eb629f6b77f P 3f1dd2529dca4aa1a50ca465d03824fd: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.001s
I20260924 16:05:33.818590 5611 tablet_replica.cc:330] T da2cae344def4201877c4426c949dd0c P 3f1dd2529dca4aa1a50ca465d03824fd: stopping tablet replica
I20260924 16:05:33.819145 5611 raft_consensus.cc:2229] T da2cae344def4201877c4426c949dd0c P 3f1dd2529dca4aa1a50ca465d03824fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:05:33.819410 5586 raft_consensus.cc:348] T af72740ecfc6424ab3ae6eb629f6b77f P 3f1dd2529dca4aa1a50ca465d03824fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } }
I20260924 16:05:33.819676 5586 raft_consensus.cc:374] T af72740ecfc6424ab3ae6eb629f6b77f P 3f1dd2529dca4aa1a50ca465d03824fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:33.821338 5586 raft_consensus.cc:729] T af72740ecfc6424ab3ae6eb629f6b77f P 3f1dd2529dca4aa1a50ca465d03824fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1dd2529dca4aa1a50ca465d03824fd, State: Initialized, Role: FOLLOWER
I20260924 16:05:33.821718 5586 consensus_queue.cc:260] T af72740ecfc6424ab3ae6eb629f6b77f P 3f1dd2529dca4aa1a50ca465d03824fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } }
I20260924 16:05:33.822590 5586 ts_tablet_manager.cc:1428] T af72740ecfc6424ab3ae6eb629f6b77f P 3f1dd2529dca4aa1a50ca465d03824fd: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 16:05:33.823254 5611 raft_consensus.cc:2258] T da2cae344def4201877c4426c949dd0c P 3f1dd2529dca4aa1a50ca465d03824fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:33.828279 5588 tablet_bootstrap.cc:492] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550: No bootstrap required, opened a new log
I20260924 16:05:33.828491 5588 ts_tablet_manager.cc:1397] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260924 16:05:33.829030 5613 ts_tablet_manager.cc:1905] T da2cae344def4201877c4426c949dd0c P a7ccd1b3730a483cb15938d6acc17936: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:33.829675 5611 ts_tablet_manager.cc:1905] T da2cae344def4201877c4426c949dd0c P 3f1dd2529dca4aa1a50ca465d03824fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:33.829574 5588 raft_consensus.cc:348] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } }
I20260924 16:05:33.829821 5588 raft_consensus.cc:374] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:33.829947 5588 raft_consensus.cc:729] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d152c4aaa6741cd9899e31b4137b550, State: Initialized, Role: FOLLOWER
I20260924 16:05:33.830336 5588 consensus_queue.cc:260] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } }
I20260924 16:05:33.831035 5588 ts_tablet_manager.cc:1428] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 16:05:33.832245 5587 tablet_bootstrap.cc:492] T af72740ecfc6424ab3ae6eb629f6b77f P a7ccd1b3730a483cb15938d6acc17936: No bootstrap required, opened a new log
I20260924 16:05:33.832442 5587 ts_tablet_manager.cc:1397] T af72740ecfc6424ab3ae6eb629f6b77f P a7ccd1b3730a483cb15938d6acc17936: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.008s
I20260924 16:05:33.833408 5587 raft_consensus.cc:348] T af72740ecfc6424ab3ae6eb629f6b77f P a7ccd1b3730a483cb15938d6acc17936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } }
I20260924 16:05:33.833671 5587 raft_consensus.cc:374] T af72740ecfc6424ab3ae6eb629f6b77f P a7ccd1b3730a483cb15938d6acc17936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:33.833781 5587 raft_consensus.cc:729] T af72740ecfc6424ab3ae6eb629f6b77f P a7ccd1b3730a483cb15938d6acc17936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ccd1b3730a483cb15938d6acc17936, State: Initialized, Role: FOLLOWER
I20260924 16:05:33.834082 5587 consensus_queue.cc:260] T af72740ecfc6424ab3ae6eb629f6b77f P a7ccd1b3730a483cb15938d6acc17936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } }
I20260924 16:05:33.834736 5587 ts_tablet_manager.cc:1428] T af72740ecfc6424ab3ae6eb629f6b77f P a7ccd1b3730a483cb15938d6acc17936: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 16:05:33.839192 5613 ts_tablet_manager.cc:1918] T da2cae344def4201877c4426c949dd0c P a7ccd1b3730a483cb15938d6acc17936: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:05:33.839440 5613 log.cc:1196] T da2cae344def4201877c4426c949dd0c P a7ccd1b3730a483cb15938d6acc17936: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/da2cae344def4201877c4426c949dd0c
I20260924 16:05:33.840099 5613 ts_tablet_manager.cc:1939] T da2cae344def4201877c4426c949dd0c P a7ccd1b3730a483cb15938d6acc17936: Deleting consensus metadata
I20260924 16:05:33.842473 5611 ts_tablet_manager.cc:1918] T da2cae344def4201877c4426c949dd0c P 3f1dd2529dca4aa1a50ca465d03824fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:05:33.842659 5611 log.cc:1196] T da2cae344def4201877c4426c949dd0c P 3f1dd2529dca4aa1a50ca465d03824fd: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/da2cae344def4201877c4426c949dd0c
I20260924 16:05:33.842617 5059 catalog_manager.cc:4841] TS a7ccd1b3730a483cb15938d6acc17936 (127.3.214.194:43303): tablet da2cae344def4201877c4426c949dd0c (table test-table [id=d89df375b14340adbd6aa4b0bddb0a65]) successfully deleted
I20260924 16:05:33.843132 5611 ts_tablet_manager.cc:1939] T da2cae344def4201877c4426c949dd0c P 3f1dd2529dca4aa1a50ca465d03824fd: Deleting consensus metadata
I20260924 16:05:33.844880 5058 catalog_manager.cc:4841] TS 3f1dd2529dca4aa1a50ca465d03824fd (127.3.214.195:40453): tablet da2cae344def4201877c4426c949dd0c (table test-table [id=d89df375b14340adbd6aa4b0bddb0a65]) successfully deleted
I20260924 16:05:33.877681 5606 raft_consensus.cc:482] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:05:33.878064 5606 raft_consensus.cc:504] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } }
I20260924 16:05:33.879262 5606 leader_election.cc:288] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7ccd1b3730a483cb15938d6acc17936 (127.3.214.194:43303), 3f1dd2529dca4aa1a50ca465d03824fd (127.3.214.195:40453)
I20260924 16:05:33.879925 5515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af72740ecfc6424ab3ae6eb629f6b77f" candidate_uuid: "8d152c4aaa6741cd9899e31b4137b550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" is_pre_election: true
I20260924 16:05:33.880030 5373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af72740ecfc6424ab3ae6eb629f6b77f" candidate_uuid: "8d152c4aaa6741cd9899e31b4137b550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7ccd1b3730a483cb15938d6acc17936" is_pre_election: true
I20260924 16:05:33.880215 5515 raft_consensus.cc:2454] T af72740ecfc6424ab3ae6eb629f6b77f P 3f1dd2529dca4aa1a50ca465d03824fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d152c4aaa6741cd9899e31b4137b550 in term 0.
I20260924 16:05:33.880345 5373 raft_consensus.cc:2454] T af72740ecfc6424ab3ae6eb629f6b77f P a7ccd1b3730a483cb15938d6acc17936 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d152c4aaa6741cd9899e31b4137b550 in term 0.
I20260924 16:05:33.880681 5163 leader_election.cc:302] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f1dd2529dca4aa1a50ca465d03824fd, 8d152c4aaa6741cd9899e31b4137b550; no voters:
I20260924 16:05:33.881057 5606 raft_consensus.cc:2781] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:05:33.881206 5606 raft_consensus.cc:482] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:05:33.881305 5606 raft_consensus.cc:3037] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:33.885155 5606 raft_consensus.cc:504] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } }
I20260924 16:05:33.886057 5606 leader_election.cc:288] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550 [CANDIDATE]: Term 1 election: Requested vote from peers a7ccd1b3730a483cb15938d6acc17936 (127.3.214.194:43303), 3f1dd2529dca4aa1a50ca465d03824fd (127.3.214.195:40453)
I20260924 16:05:33.886870 5515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af72740ecfc6424ab3ae6eb629f6b77f" candidate_uuid: "8d152c4aaa6741cd9899e31b4137b550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1dd2529dca4aa1a50ca465d03824fd"
I20260924 16:05:33.887164 5515 raft_consensus.cc:3037] T af72740ecfc6424ab3ae6eb629f6b77f P 3f1dd2529dca4aa1a50ca465d03824fd [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:33.890568 5515 raft_consensus.cc:2454] T af72740ecfc6424ab3ae6eb629f6b77f P 3f1dd2529dca4aa1a50ca465d03824fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d152c4aaa6741cd9899e31b4137b550 in term 1.
I20260924 16:05:33.890530 5373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af72740ecfc6424ab3ae6eb629f6b77f" candidate_uuid: "8d152c4aaa6741cd9899e31b4137b550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7ccd1b3730a483cb15938d6acc17936"
I20260924 16:05:33.890818 5373 raft_consensus.cc:3037] T af72740ecfc6424ab3ae6eb629f6b77f P a7ccd1b3730a483cb15938d6acc17936 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:33.891294 5163 leader_election.cc:302] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f1dd2529dca4aa1a50ca465d03824fd, 8d152c4aaa6741cd9899e31b4137b550; no voters:
I20260924 16:05:33.891786 5606 raft_consensus.cc:2781] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:33.892139 5606 raft_consensus.cc:686] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550 [term 1 LEADER]: Becoming Leader. State: Replica: 8d152c4aaa6741cd9899e31b4137b550, State: Running, Role: LEADER
I20260924 16:05:33.892479 5606 consensus_queue.cc:237] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } }
I20260924 16:05:33.893989 5373 raft_consensus.cc:2454] T af72740ecfc6424ab3ae6eb629f6b77f P a7ccd1b3730a483cb15938d6acc17936 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d152c4aaa6741cd9899e31b4137b550 in term 1.
I20260924 16:05:33.896649 5073 catalog_manager.cc:5495] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d152c4aaa6741cd9899e31b4137b550 (127.3.214.193). New cstate: current_term: 1 leader_uuid: "8d152c4aaa6741cd9899e31b4137b550" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } health_report { overall_health: UNKNOWN } } }
Sep 24 16:05:33 dist-test-slave-1gc3 krb5kdc[5029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265931, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:33.965457 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:33.949018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39961 (local address 127.0.0.1:50402)
0924 16:05:33.949791 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:33.949810 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:33.949986 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:33.950385 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:33.950390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:33.951154 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:33.951431 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:33.951441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.952579 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.952584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:33.953562 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:33.953573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.954319 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.954325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:33.954490 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:33.955320 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:33.955339 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:33.958316 (+ 2977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:33.961786 (+ 3470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:33.961812 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:33.961824 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:33.962668 (+ 844us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:33.963163 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:33.963168 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:33.963172 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:33.963379 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:33.963994 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:33.964006 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:33.964278 (+ 272us) client_negotiation.cc:769] Sending connection context
0924 16:05:33.964490 (+ 212us) client_negotiation.cc:240] Negotiation successful
0924 16:05:33.964953 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":476,"threads_started":1}
I20260924 16:05:33.965559 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:33.949011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50402 (local address 127.3.214.254:39961)
0924 16:05:33.949715 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:33.949721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:33.949909 (+ 188us) server_negotiation.cc:407] Connection header received
0924 16:05:33.950132 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:33.950137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:33.950213 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:33.950306 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:33.951598 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.952447 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.953698 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:33.954152 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:33.954269 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:33.958675 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:33.958682 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:33.958686 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:33.958728 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:33.961570 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:33.962863 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:33.962868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:33.962870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:33.963030 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:33.963501 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:33.963505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:33.963507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:33.963849 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:33.963981 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:33.964654 (+ 673us) server_negotiation.cc:299] Negotiation successful
0924 16:05:33.965166 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":175,"server-negotiator.queue_time_us":520,"thread_start_us":277,"threads_started":1}
I20260924 16:05:33.971756 5073 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:50402:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 16:05:33.972980 5073 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:05:33.981385 5213 tablet_service.cc:1467] Processing CreateTablet for tablet c559f235a34a40789b4ca9f85e6d920c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=acadd1c020d24027941f37e7505d5a13]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:05:33.982182 5213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c559f235a34a40789b4ca9f85e6d920c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:33.985553 5495 tablet_service.cc:1467] Processing CreateTablet for tablet c559f235a34a40789b4ca9f85e6d920c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=acadd1c020d24027941f37e7505d5a13]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:05:33.986240 5495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c559f235a34a40789b4ca9f85e6d920c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:33.986802 5353 tablet_service.cc:1467] Processing CreateTablet for tablet c559f235a34a40789b4ca9f85e6d920c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=acadd1c020d24027941f37e7505d5a13]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:05:33.987401 5353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c559f235a34a40789b4ca9f85e6d920c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:33.994709 5586 tablet_bootstrap.cc:492] T c559f235a34a40789b4ca9f85e6d920c P 3f1dd2529dca4aa1a50ca465d03824fd: Bootstrap starting.
I20260924 16:05:33.997170 5586 tablet_bootstrap.cc:654] T c559f235a34a40789b4ca9f85e6d920c P 3f1dd2529dca4aa1a50ca465d03824fd: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:34.000036 5587 tablet_bootstrap.cc:492] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936: Bootstrap starting.
I20260924 16:05:34.002344 5587 tablet_bootstrap.cc:654] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:34.004066 5586 tablet_bootstrap.cc:492] T c559f235a34a40789b4ca9f85e6d920c P 3f1dd2529dca4aa1a50ca465d03824fd: No bootstrap required, opened a new log
I20260924 16:05:34.004455 5586 ts_tablet_manager.cc:1397] T c559f235a34a40789b4ca9f85e6d920c P 3f1dd2529dca4aa1a50ca465d03824fd: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260924 16:05:34.005383 5586 raft_consensus.cc:348] T c559f235a34a40789b4ca9f85e6d920c P 3f1dd2529dca4aa1a50ca465d03824fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } }
I20260924 16:05:34.005764 5586 raft_consensus.cc:374] T c559f235a34a40789b4ca9f85e6d920c P 3f1dd2529dca4aa1a50ca465d03824fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:34.005826 5588 tablet_bootstrap.cc:492] T c559f235a34a40789b4ca9f85e6d920c P 8d152c4aaa6741cd9899e31b4137b550: Bootstrap starting.
I20260924 16:05:34.005937 5586 raft_consensus.cc:729] T c559f235a34a40789b4ca9f85e6d920c P 3f1dd2529dca4aa1a50ca465d03824fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1dd2529dca4aa1a50ca465d03824fd, State: Initialized, Role: FOLLOWER
I20260924 16:05:34.006390 5586 consensus_queue.cc:260] T c559f235a34a40789b4ca9f85e6d920c P 3f1dd2529dca4aa1a50ca465d03824fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } }
I20260924 16:05:34.006981 5593 tablet_replica.cc:439] T c559f235a34a40789b4ca9f85e6d920c P 3f1dd2529dca4aa1a50ca465d03824fd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } } }
I20260924 16:05:34.007316 5593 tablet_replica.cc:442] T c559f235a34a40789b4ca9f85e6d920c P 3f1dd2529dca4aa1a50ca465d03824fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:05:34.007936 5588 tablet_bootstrap.cc:654] T c559f235a34a40789b4ca9f85e6d920c P 8d152c4aaa6741cd9899e31b4137b550: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:34.011098 5586 ts_tablet_manager.cc:1428] T c559f235a34a40789b4ca9f85e6d920c P 3f1dd2529dca4aa1a50ca465d03824fd: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260924 16:05:34.011389 5587 tablet_bootstrap.cc:492] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936: No bootstrap required, opened a new log
I20260924 16:05:34.011861 5587 ts_tablet_manager.cc:1397] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.005s
I20260924 16:05:34.012867 5587 raft_consensus.cc:348] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } }
I20260924 16:05:34.013175 5587 raft_consensus.cc:374] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:34.013928 5587 raft_consensus.cc:729] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ccd1b3730a483cb15938d6acc17936, State: Initialized, Role: FOLLOWER
I20260924 16:05:34.014364 5587 consensus_queue.cc:260] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } }
I20260924 16:05:34.015120 5587 ts_tablet_manager.cc:1428] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260924 16:05:34.015071 5592 tablet_replica.cc:439] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } } }
I20260924 16:05:34.015693 5592 tablet_replica.cc:442] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:05:34.018658 5588 tablet_bootstrap.cc:492] T c559f235a34a40789b4ca9f85e6d920c P 8d152c4aaa6741cd9899e31b4137b550: No bootstrap required, opened a new log
I20260924 16:05:34.018859 5588 ts_tablet_manager.cc:1397] T c559f235a34a40789b4ca9f85e6d920c P 8d152c4aaa6741cd9899e31b4137b550: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260924 16:05:34.019531 5588 raft_consensus.cc:348] T c559f235a34a40789b4ca9f85e6d920c P 8d152c4aaa6741cd9899e31b4137b550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } }
I20260924 16:05:34.019757 5588 raft_consensus.cc:374] T c559f235a34a40789b4ca9f85e6d920c P 8d152c4aaa6741cd9899e31b4137b550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:34.019843 5588 raft_consensus.cc:729] T c559f235a34a40789b4ca9f85e6d920c P 8d152c4aaa6741cd9899e31b4137b550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d152c4aaa6741cd9899e31b4137b550, State: Initialized, Role: FOLLOWER
I20260924 16:05:34.020105 5588 consensus_queue.cc:260] T c559f235a34a40789b4ca9f85e6d920c P 8d152c4aaa6741cd9899e31b4137b550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } }
W20260924 16:05:34.020417 5425 tablet.cc:2369] T af72740ecfc6424ab3ae6eb629f6b77f P a7ccd1b3730a483cb15938d6acc17936: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:05:34.020632 5606 tablet_replica.cc:439] T c559f235a34a40789b4ca9f85e6d920c P 8d152c4aaa6741cd9899e31b4137b550: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } } }
I20260924 16:05:34.020901 5588 ts_tablet_manager.cc:1428] T c559f235a34a40789b4ca9f85e6d920c P 8d152c4aaa6741cd9899e31b4137b550: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 16:05:34.021035 5606 tablet_replica.cc:442] T c559f235a34a40789b4ca9f85e6d920c P 8d152c4aaa6741cd9899e31b4137b550: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:05:34.047847 5592 raft_consensus.cc:482] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:05:34.048159 5592 raft_consensus.cc:504] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } }
I20260924 16:05:34.051992 5592 leader_election.cc:288] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d152c4aaa6741cd9899e31b4137b550 (127.3.214.193:35793), 3f1dd2529dca4aa1a50ca465d03824fd (127.3.214.195:40453)
I20260924 16:05:34.059746 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:34.051983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:37551 (local address 127.3.214.195:40453)
0924 16:05:34.052191 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:34.052196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:34.053260 (+ 1064us) server_negotiation.cc:407] Connection header received
0924 16:05:34.053479 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:34.053485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:34.053572 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:34.053675 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:34.054482 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.055291 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.057271 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.058505 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.058690 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:34.058781 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:34.059266 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 16:05:34.059421 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 16:05:34.059901 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:34.051350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:40453 (local address 127.3.214.194:37551)
0924 16:05:34.053098 (+ 1748us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:34.053131 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:34.053332 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:34.053732 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:34.053738 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:34.053771 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:34.054270 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:34.054286 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.055455 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.055462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:34.057092 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:34.057109 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.058796 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.058803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:34.059029 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:34.059081 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 16:05:34.059211 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 16:05:34.059422 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1581,"mutex_wait_us":1,"thread_start_us":1266,"threads_started":1}
I20260924 16:05:34.060122 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:34.050682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:45753 (local address 127.3.214.193:35793)
0924 16:05:34.050879 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:34.050886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:34.051721 (+ 835us) server_negotiation.cc:407] Connection header received
0924 16:05:34.051909 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:34.051915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:34.052012 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:34.052163 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:34.053397 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.054515 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.057508 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.058858 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.059020 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:34.059103 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:34.059571 (+ 468us) server_negotiation.cc:299] Negotiation successful
0924 16:05:34.059727 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 16:05:34.060710 5628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:34.050554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:35793 (local address 127.3.214.194:45753)
0924 16:05:34.051486 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:34.051514 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:34.051722 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:34.052591 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:34.052599 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:34.052658 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:34.053169 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:34.053181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.054694 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.054700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:34.057281 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:34.057301 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.059148 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.059156 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:34.059351 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:34.059396 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 16:05:34.060194 (+ 798us) client_negotiation.cc:240] Negotiation successful
0924 16:05:34.060381 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":1671,"threads_started":1}
I20260924 16:05:34.060987 5515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c559f235a34a40789b4ca9f85e6d920c" candidate_uuid: "a7ccd1b3730a483cb15938d6acc17936" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" is_pre_election: true
I20260924 16:05:34.061331 5515 raft_consensus.cc:2454] T c559f235a34a40789b4ca9f85e6d920c P 3f1dd2529dca4aa1a50ca465d03824fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7ccd1b3730a483cb15938d6acc17936 in term 0.
I20260924 16:05:34.062144 5303 leader_election.cc:302] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f1dd2529dca4aa1a50ca465d03824fd, a7ccd1b3730a483cb15938d6acc17936; no voters:
I20260924 16:05:34.061848 5233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c559f235a34a40789b4ca9f85e6d920c" candidate_uuid: "a7ccd1b3730a483cb15938d6acc17936" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d152c4aaa6741cd9899e31b4137b550" is_pre_election: true
I20260924 16:05:34.062646 5233 raft_consensus.cc:2454] T c559f235a34a40789b4ca9f85e6d920c P 8d152c4aaa6741cd9899e31b4137b550 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7ccd1b3730a483cb15938d6acc17936 in term 0.
I20260924 16:05:34.062775 5592 raft_consensus.cc:2781] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:05:34.062906 5592 raft_consensus.cc:482] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:05:34.062992 5592 raft_consensus.cc:3037] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:34.066776 5592 raft_consensus.cc:504] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } }
I20260924 16:05:34.067780 5592 leader_election.cc:288] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936 [CANDIDATE]: Term 1 election: Requested vote from peers 8d152c4aaa6741cd9899e31b4137b550 (127.3.214.193:35793), 3f1dd2529dca4aa1a50ca465d03824fd (127.3.214.195:40453)
I20260924 16:05:34.068617 5515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c559f235a34a40789b4ca9f85e6d920c" candidate_uuid: "a7ccd1b3730a483cb15938d6acc17936" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1dd2529dca4aa1a50ca465d03824fd"
I20260924 16:05:34.068670 5233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c559f235a34a40789b4ca9f85e6d920c" candidate_uuid: "a7ccd1b3730a483cb15938d6acc17936" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d152c4aaa6741cd9899e31b4137b550"
I20260924 16:05:34.068935 5233 raft_consensus.cc:3037] T c559f235a34a40789b4ca9f85e6d920c P 8d152c4aaa6741cd9899e31b4137b550 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:34.068940 5515 raft_consensus.cc:3037] T c559f235a34a40789b4ca9f85e6d920c P 3f1dd2529dca4aa1a50ca465d03824fd [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:34.071643 5233 raft_consensus.cc:2454] T c559f235a34a40789b4ca9f85e6d920c P 8d152c4aaa6741cd9899e31b4137b550 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7ccd1b3730a483cb15938d6acc17936 in term 1.
I20260924 16:05:34.072268 5305 leader_election.cc:302] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d152c4aaa6741cd9899e31b4137b550, a7ccd1b3730a483cb15938d6acc17936; no voters:
I20260924 16:05:34.072427 5515 raft_consensus.cc:2454] T c559f235a34a40789b4ca9f85e6d920c P 3f1dd2529dca4aa1a50ca465d03824fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7ccd1b3730a483cb15938d6acc17936 in term 1.
I20260924 16:05:34.072636 5592 raft_consensus.cc:2781] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:34.074179 5592 raft_consensus.cc:686] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936 [term 1 LEADER]: Becoming Leader. State: Replica: a7ccd1b3730a483cb15938d6acc17936, State: Running, Role: LEADER
I20260924 16:05:34.074653 5592 consensus_queue.cc:237] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } }
I20260924 16:05:34.077855 5630 tablet_replica.cc:439] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936: TxnStatusTablet state changed. Reason: New leader a7ccd1b3730a483cb15938d6acc17936. Latest consensus state: current_term: 1 leader_uuid: "a7ccd1b3730a483cb15938d6acc17936" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } } }
I20260924 16:05:34.078482 5630 tablet_replica.cc:442] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:05:34.080207 5074 catalog_manager.cc:5495] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7ccd1b3730a483cb15938d6acc17936 (127.3.214.194). New cstate: current_term: 1 leader_uuid: "a7ccd1b3730a483cb15938d6acc17936" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } health_report { overall_health: UNKNOWN } } }
I20260924 16:05:34.081414 5632 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 16:05:34.081722 5632 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 16:05:34.122901 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:34.113623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59686 (local address 127.3.214.194:43303)
0924 16:05:34.113861 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:34.113869 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:34.113988 (+ 119us) server_negotiation.cc:407] Connection header received
0924 16:05:34.114390 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:34.114396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:34.114531 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:34.114707 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:34.115195 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.116724 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.119098 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.119869 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.120112 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:34.120944 (+ 832us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:34.121859 (+ 915us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:34.122071 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:34.122334 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 16:05:34.122481 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 16:05:34.122912 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:34.113562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:43303 (local address 127.0.0.1:59686)
0924 16:05:34.113888 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:34.113908 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:34.114078 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:34.114760 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:34.114766 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:34.114781 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:05:34.115043 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:34.115055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.116977 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.116985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:34.118769 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:34.118799 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.120196 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.120205 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:34.120453 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:34.120582 (+ 129us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:05:34.122065 (+ 1483us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:05:34.122143 (+ 78us) client_negotiation.cc:769] Sending connection context
0924 16:05:34.122279 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 16:05:34.122520 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 16:05:34.129001 5515 raft_consensus.cc:1261] T c559f235a34a40789b4ca9f85e6d920c P 3f1dd2529dca4aa1a50ca465d03824fd [term 1 FOLLOWER]: Refusing update from remote peer a7ccd1b3730a483cb15938d6acc17936: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:34.129665 5233 raft_consensus.cc:1261] T c559f235a34a40789b4ca9f85e6d920c P 8d152c4aaa6741cd9899e31b4137b550 [term 1 FOLLOWER]: Refusing update from remote peer a7ccd1b3730a483cb15938d6acc17936: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:34.130348 5592 consensus_queue.cc:1035] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:34.131640 5592 consensus_queue.cc:1035] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:34.139096 5593 tablet_replica.cc:439] T c559f235a34a40789b4ca9f85e6d920c P 3f1dd2529dca4aa1a50ca465d03824fd: TxnStatusTablet state changed. Reason: New leader a7ccd1b3730a483cb15938d6acc17936. Latest consensus state: current_term: 1 leader_uuid: "a7ccd1b3730a483cb15938d6acc17936" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } } }
I20260924 16:05:34.139428 5593 tablet_replica.cc:442] T c559f235a34a40789b4ca9f85e6d920c P 3f1dd2529dca4aa1a50ca465d03824fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:05:34.144636 5592 tablet_replica.cc:439] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7ccd1b3730a483cb15938d6acc17936" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } } }
I20260924 16:05:34.145136 5592 tablet_replica.cc:442] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:05:34.148974 5602 mvcc.cc:205] Tried to move back new op lower bound from 7332929266182119424 to 7332929265975586816. Current Snapshot: MvccSnapshot[applied={T|T < 7332929266182119424}]
I20260924 16:05:34.150588 5593 tablet_replica.cc:439] T c559f235a34a40789b4ca9f85e6d920c P 3f1dd2529dca4aa1a50ca465d03824fd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7ccd1b3730a483cb15938d6acc17936" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } } }
I20260924 16:05:34.150926 5593 tablet_replica.cc:442] T c559f235a34a40789b4ca9f85e6d920c P 3f1dd2529dca4aa1a50ca465d03824fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:05:34.158439 5606 tablet_replica.cc:439] T c559f235a34a40789b4ca9f85e6d920c P 8d152c4aaa6741cd9899e31b4137b550: TxnStatusTablet state changed. Reason: New leader a7ccd1b3730a483cb15938d6acc17936. Latest consensus state: current_term: 1 leader_uuid: "a7ccd1b3730a483cb15938d6acc17936" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } } }
I20260924 16:05:34.158883 5606 tablet_replica.cc:442] T c559f235a34a40789b4ca9f85e6d920c P 8d152c4aaa6741cd9899e31b4137b550: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:05:34.164305 5630 tablet_replica.cc:439] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7ccd1b3730a483cb15938d6acc17936" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } } }
I20260924 16:05:34.164711 5630 tablet_replica.cc:442] T c559f235a34a40789b4ca9f85e6d920c P a7ccd1b3730a483cb15938d6acc17936: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:05:34.167994 5606 tablet_replica.cc:439] T c559f235a34a40789b4ca9f85e6d920c P 8d152c4aaa6741cd9899e31b4137b550: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7ccd1b3730a483cb15938d6acc17936" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d152c4aaa6741cd9899e31b4137b550" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 35793 } } peers { permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 } } peers { permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 } } }
I20260924 16:05:34.168380 5606 tablet_replica.cc:442] T c559f235a34a40789b4ca9f85e6d920c P 8d152c4aaa6741cd9899e31b4137b550: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:05:34.216986 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:34.208820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:60993 (local address 127.3.214.194:43303)
0924 16:05:34.209140 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:34.209148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:34.209720 (+ 572us) server_negotiation.cc:407] Connection header received
0924 16:05:34.210008 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:34.210015 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:34.210163 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:34.210298 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:34.210983 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.212178 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.214485 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.215194 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.215341 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:34.216087 (+ 746us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:34.216254 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:34.216354 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:34.216550 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 16:05:34.216649 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260924 16:05:34.217501 5639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:34.208686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:43303 (local address 127.3.214.193:60993)
0924 16:05:34.209547 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:34.209578 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:34.209832 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:34.210354 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:34.210359 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:34.210374 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:05:34.210773 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:34.210788 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.212367 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.212374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:34.214292 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:34.214308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.215455 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.215467 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:34.215695 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:34.215823 (+ 128us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:05:34.216390 (+ 567us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:05:34.216404 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 16:05:34.216509 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 16:05:34.216691 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":457,"threads_started":1}
I20260924 16:05:34.234807 5515 raft_consensus.cc:1261] T af72740ecfc6424ab3ae6eb629f6b77f P 3f1dd2529dca4aa1a50ca465d03824fd [term 1 FOLLOWER]: Refusing update from remote peer 8d152c4aaa6741cd9899e31b4137b550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:34.235143 5373 raft_consensus.cc:1261] T af72740ecfc6424ab3ae6eb629f6b77f P a7ccd1b3730a483cb15938d6acc17936 [term 1 FOLLOWER]: Refusing update from remote peer 8d152c4aaa6741cd9899e31b4137b550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:34.235620 5594 consensus_queue.cc:1035] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1dd2529dca4aa1a50ca465d03824fd" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 40453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:34.236083 5606 consensus_queue.cc:1035] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7ccd1b3730a483cb15938d6acc17936" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:34.305124 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:34.294826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:42991 (local address 127.3.214.193:35793)
0924 16:05:34.295087 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:34.295094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:34.296306 (+ 1212us) server_negotiation.cc:407] Connection header received
0924 16:05:34.296648 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:34.296652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:34.296748 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:34.296831 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:34.297638 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.298646 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.302001 (+ 3355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.302596 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.302740 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:34.303391 (+ 651us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:34.303573 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:34.303641 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:34.303891 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 16:05:34.304008 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 16:05:34.305668 5644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:34.295310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:35793 (local address 127.3.214.194:42991)
0924 16:05:34.296159 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:34.296184 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:34.296470 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:34.296873 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:34.296878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:34.296891 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:05:34.297406 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:34.297426 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.299422 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.299428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:34.301792 (+ 2364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:34.301815 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.302820 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.302826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:34.303019 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:34.303122 (+ 103us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:05:34.303731 (+ 609us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:05:34.303743 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 16:05:34.304450 (+ 707us) client_negotiation.cc:240] Negotiation successful
0924 16:05:34.304730 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":414,"threads_started":1}
I20260924 16:05:34.360235 5072 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:05:34.360565 5072 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:05:34.365020 5072 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f00980684834b2cb732598908639eae: Sending DeleteTablet for 3 replicas of tablet af72740ecfc6424ab3ae6eb629f6b77f
I20260924 16:05:34.366205 5213 tablet_service.cc:1514] Processing DeleteTablet for tablet af72740ecfc6424ab3ae6eb629f6b77f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:34 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:57908
I20260924 16:05:34.366525 5495 tablet_service.cc:1514] Processing DeleteTablet for tablet af72740ecfc6424ab3ae6eb629f6b77f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:34 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:48140
I20260924 16:05:34.367295 5353 tablet_service.cc:1514] Processing DeleteTablet for tablet af72740ecfc6424ab3ae6eb629f6b77f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:34 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:59670
I20260924 16:05:34.377091 5646 tablet_replica.cc:330] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550: stopping tablet replica
I20260924 16:05:34.377391 5646 raft_consensus.cc:2229] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:05:34.377844 5646 raft_consensus.cc:2258] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:34.378619 5648 tablet_replica.cc:330] T af72740ecfc6424ab3ae6eb629f6b77f P a7ccd1b3730a483cb15938d6acc17936: stopping tablet replica
I20260924 16:05:34.378911 5648 raft_consensus.cc:2229] T af72740ecfc6424ab3ae6eb629f6b77f P a7ccd1b3730a483cb15938d6acc17936 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:05:34.379230 5648 raft_consensus.cc:2258] T af72740ecfc6424ab3ae6eb629f6b77f P a7ccd1b3730a483cb15938d6acc17936 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:34.387373 5646 ts_tablet_manager.cc:1905] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:34.389098 5648 ts_tablet_manager.cc:1905] T af72740ecfc6424ab3ae6eb629f6b77f P a7ccd1b3730a483cb15938d6acc17936: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:34.397374 5646 ts_tablet_manager.cc:1918] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:05:34.397568 5646 log.cc:1196] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-0/wal/wals/af72740ecfc6424ab3ae6eb629f6b77f
I20260924 16:05:34.399235 5647 tablet_replica.cc:330] T af72740ecfc6424ab3ae6eb629f6b77f P 3f1dd2529dca4aa1a50ca465d03824fd: stopping tablet replica
I20260924 16:05:34.399506 5647 raft_consensus.cc:2229] T af72740ecfc6424ab3ae6eb629f6b77f P 3f1dd2529dca4aa1a50ca465d03824fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:05:34.399850 5647 raft_consensus.cc:2258] T af72740ecfc6424ab3ae6eb629f6b77f P 3f1dd2529dca4aa1a50ca465d03824fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:34.398048 5646 ts_tablet_manager.cc:1939] T af72740ecfc6424ab3ae6eb629f6b77f P 8d152c4aaa6741cd9899e31b4137b550: Deleting consensus metadata
I20260924 16:05:34.401448 5060 catalog_manager.cc:4841] TS 8d152c4aaa6741cd9899e31b4137b550 (127.3.214.193:35793): tablet af72740ecfc6424ab3ae6eb629f6b77f (table test-table [id=f6a0c50669584a62a6da304af022a73a]) successfully deleted
I20260924 16:05:34.402518 5647 ts_tablet_manager.cc:1905] T af72740ecfc6424ab3ae6eb629f6b77f P 3f1dd2529dca4aa1a50ca465d03824fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:34.398016 5648 ts_tablet_manager.cc:1918] T af72740ecfc6424ab3ae6eb629f6b77f P a7ccd1b3730a483cb15938d6acc17936: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:05:34.405102 5648 log.cc:1196] T af72740ecfc6424ab3ae6eb629f6b77f P a7ccd1b3730a483cb15938d6acc17936: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/af72740ecfc6424ab3ae6eb629f6b77f
I20260924 16:05:34.405764 5648 ts_tablet_manager.cc:1939] T af72740ecfc6424ab3ae6eb629f6b77f P a7ccd1b3730a483cb15938d6acc17936: Deleting consensus metadata
I20260924 16:05:34.407456 5059 catalog_manager.cc:4841] TS a7ccd1b3730a483cb15938d6acc17936 (127.3.214.194:43303): tablet af72740ecfc6424ab3ae6eb629f6b77f (table test-table [id=f6a0c50669584a62a6da304af022a73a]) successfully deleted
I20260924 16:05:34.410913 5647 ts_tablet_manager.cc:1918] T af72740ecfc6424ab3ae6eb629f6b77f P 3f1dd2529dca4aa1a50ca465d03824fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:05:34.411108 5647 log.cc:1196] T af72740ecfc6424ab3ae6eb629f6b77f P 3f1dd2529dca4aa1a50ca465d03824fd: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/af72740ecfc6424ab3ae6eb629f6b77f
I20260924 16:05:34.411561 5647 ts_tablet_manager.cc:1939] T af72740ecfc6424ab3ae6eb629f6b77f P 3f1dd2529dca4aa1a50ca465d03824fd: Deleting consensus metadata
Sep 24 16:05:34 dist-test-slave-1gc3 krb5kdc[5029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265933, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.214.193@KRBTEST.COM
I20260924 16:05:34.412984 5058 catalog_manager.cc:4841] TS 3f1dd2529dca4aa1a50ca465d03824fd (127.3.214.195:40453): tablet af72740ecfc6424ab3ae6eb629f6b77f (table test-table [id=f6a0c50669584a62a6da304af022a73a]) successfully deleted
I20260924 16:05:34.421300 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:34.390660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57928 (local address 127.3.214.193:35793)
0924 16:05:34.390935 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:34.390942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:34.398955 (+ 8013us) server_negotiation.cc:407] Connection header received
0924 16:05:34.399296 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:34.399303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:34.399408 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:34.399496 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:34.401745 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.402889 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.407879 (+ 4990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.408611 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.408723 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:34.414297 (+ 5574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:34.414322 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:34.414348 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:34.414401 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:34.417923 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:34.418882 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:34.418891 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:34.418894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:34.419059 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:34.419464 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:34.419469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:34.419471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:34.419943 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:34.420278 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:34.420626 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 16:05:34.420918 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 16:05:34.422284 5278 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57928
I20260924 16:05:34.453684 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:34.425930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50412 (local address 127.3.214.254:39961)
0924 16:05:34.426280 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:34.426288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:34.438958 (+ 12670us) server_negotiation.cc:407] Connection header received
0924 16:05:34.439176 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:34.439181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:34.439280 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:34.439377 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:34.440933 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.441849 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.443465 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.444036 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.444158 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:34.447123 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:34.447128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:34.447132 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:34.447177 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:34.450644 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:34.451529 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:34.451534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:34.451537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:34.451678 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:34.452078 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:34.452082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:34.452084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:34.452429 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:34.452689 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:34.453064 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 16:05:34.453341 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260924 16:05:34.454480 5072 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50412
I20260924 16:05:34.457379 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 5146
I20260924 16:05:34.472734 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:34.458736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57936 (local address 127.3.214.193:35793)
0924 16:05:34.458936 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:34.458943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:34.459296 (+ 353us) server_negotiation.cc:407] Connection header received
0924 16:05:34.459501 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:34.459505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:34.459585 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:34.459682 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:34.460955 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.461750 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.463313 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.463762 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.463881 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:34.466689 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:34.466694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:34.466697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:34.466740 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:34.469779 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:34.470574 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:34.470579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:34.470582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:34.470750 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:34.471178 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:34.471182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:34.471185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:34.471514 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:34.471784 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:34.472098 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 16:05:34.472367 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 16:05:34.473480 5278 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57936
W20260924 16:05:34.474189 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 5146: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:05:34.474393 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 5146
I20260924 16:05:34.504642 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 5287
Sep 24 16:05:34 dist-test-slave-1gc3 krb5kdc[5029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265933, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.214.194@KRBTEST.COM
I20260924 16:05:34.524194 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:34.505602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59690 (local address 127.3.214.194:43303)
0924 16:05:34.505887 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:34.505894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:34.505996 (+ 102us) server_negotiation.cc:407] Connection header received
0924 16:05:34.506256 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:34.506262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:34.506375 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:34.506472 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:34.507815 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.508758 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.510212 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.510825 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.510967 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:34.516151 (+ 5184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:34.516173 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:34.516202 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:34.516284 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:34.520346 (+ 4062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:34.521305 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:34.521314 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:34.521317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:34.521536 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:34.522337 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:34.522342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:34.522345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:34.522824 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:34.523116 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:34.523596 (+ 480us) server_negotiation.cc:299] Negotiation successful
0924 16:05:34.523827 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260924 16:05:34.525157 5419 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59690
W20260924 16:05:34.526144 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 5287: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:05:34.526407 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 5287
I20260924 16:05:34.550966 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 5428
Sep 24 16:05:34 dist-test-slave-1gc3 krb5kdc[5029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265933, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.214.195@KRBTEST.COM
I20260924 16:05:34.568329 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:34.551864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51976 (local address 127.3.214.195:40453)
0924 16:05:34.552120 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:34.552128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:34.552206 (+ 78us) server_negotiation.cc:407] Connection header received
0924 16:05:34.552421 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:34.552426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:34.552524 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:34.552626 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:34.554023 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.554774 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.556057 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:34.556646 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:34.556786 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:34.562083 (+ 5297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:34.562245 (+ 162us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:34.562268 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:34.562315 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:34.565270 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:34.566064 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:34.566072 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:34.566075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:34.566267 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:34.566676 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:34.566680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:34.566682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:34.567086 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:34.567383 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:34.567724 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 16:05:34.567973 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260924 16:05:34.569265 5562 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51976
W20260924 16:05:34.570022 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 5428: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:05:34.570235 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 5428
I20260924 16:05:34.588253 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 5042
I20260924 16:05:34.589779 5115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:34.719542 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 5042
2026-09-24T16:05:34Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5059 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:05:34 dist-test-slave-1gc3 krb5kdc[5658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:05:34 dist-test-slave-1gc3 krb5kdc[5658](info): set up 2 sockets
Sep 24 16:05:34 dist-test-slave-1gc3 krb5kdc[5658](info): commencing operation
krb5kdc: starting...
W20260924 16:05:35.751926 3931 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:05:35 dist-test-slave-1gc3 krb5kdc[5658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:05:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:05:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:05:35.973977 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:43765
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:43765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:36.377491 5671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:36.377897 5671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:36.378046 5671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:36.386598 5671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:05:36.386780 5671 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 16:05:36.386855 5671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:36.386948 5671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:36.387022 5671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:05:36.387142 5671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:05:36.399106 5671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:43765
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:43765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:36.400907 5671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:36.402667 5671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:36.417304 5676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:36.417409 5677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:36.417335 5679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:36.418543 5671 server_base.cc:1034] running on GCE node
I20260924 16:05:36.419605 5671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:36.421528 5671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:36.422838 5671 hybrid_clock.cc:648] HybridClock initialized: now 1790265936422727 us; error 67 us; skew 500 ppm
I20260924 16:05:36.423349 5671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:36 dist-test-slave-1gc3 krb5kdc[5658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265936, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:36.427639 5671 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:05:36.430025 5671 webserver.cc:460] Webserver started at http://127.3.214.254:37081/ using document root <none> and password file <none>
I20260924 16:05:36.431149 5671 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:36.431301 5671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:36.431700 5671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:36.436183 5671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "97c89026d44e402093c6635a028db884"
format_stamp: "Formatted at 2026-09-24 16:05:36 on dist-test-slave-1gc3"
server_key: "f3d0a311bcb2dd08cb224a6f2bdf7aaa"
server_key_iv: "88bab2216d4bae3fc3b02977b8523d25"
server_key_version: "encryptionkey@0"
I20260924 16:05:36.437295 5671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "97c89026d44e402093c6635a028db884"
format_stamp: "Formatted at 2026-09-24 16:05:36 on dist-test-slave-1gc3"
server_key: "f3d0a311bcb2dd08cb224a6f2bdf7aaa"
server_key_iv: "88bab2216d4bae3fc3b02977b8523d25"
server_key_version: "encryptionkey@0"
I20260924 16:05:36.445318 5671 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.006s
I20260924 16:05:36.451231 5686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:36.453306 5671 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260924 16:05:36.453589 5671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "97c89026d44e402093c6635a028db884"
format_stamp: "Formatted at 2026-09-24 16:05:36 on dist-test-slave-1gc3"
server_key: "f3d0a311bcb2dd08cb224a6f2bdf7aaa"
server_key_iv: "88bab2216d4bae3fc3b02977b8523d25"
server_key_version: "encryptionkey@0"
I20260924 16:05:36.453902 5671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:36.486171 5671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:36.491936 5671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:36.492264 5671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:36.518256 5671 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:43765
I20260924 16:05:36.518278 5749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:43765 every 8 connection(s)
I20260924 16:05:36.520100 5671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:05:36.520498 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 5671
I20260924 16:05:36.520717 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:05:36.521471 3931 external_mini_cluster.cc:1428] Setting key d9fa893b9698f722e108604501f55080
I20260924 16:05:36.526240 5750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 16:05:36 dist-test-slave-1gc3 krb5kdc[5658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:36.539783 5750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884: Bootstrap starting.
I20260924 16:05:36.544382 5750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:36.545888 5750 log.cc:824] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:36.548120 5753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:36.523980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43838 (local address 127.3.214.254:43765)
0924 16:05:36.525016 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:36.525034 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:36.525092 (+ 58us) server_negotiation.cc:407] Connection header received
0924 16:05:36.527155 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:36.527190 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:36.527729 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:36.528155 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:36.529784 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:36.531357 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:36.532696 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:36.533496 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:36.533703 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:36.538371 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:36.538391 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:36.538414 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:36.538472 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:36.542521 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:36.543423 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:36.543432 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:36.543441 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:36.543666 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:36.544078 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:36.544082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:36.544084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:36.544886 (+ 802us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:36.545234 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:36.545880 (+ 646us) server_negotiation.cc:299] Negotiation successful
0924 16:05:36.546535 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":462,"threads_started":1}
I20260924 16:05:36.550068 5750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884: No bootstrap required, opened a new log
I20260924 16:05:36.559137 5750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c89026d44e402093c6635a028db884" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43765 } }
I20260924 16:05:36.559672 5750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:36.559777 5750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c89026d44e402093c6635a028db884, State: Initialized, Role: FOLLOWER
I20260924 16:05:36.560489 5750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c89026d44e402093c6635a028db884" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43765 } }
I20260924 16:05:36.560761 5750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:05:36.560909 5750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:05:36.561100 5750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:36.564000 5750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c89026d44e402093c6635a028db884" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43765 } }
I20260924 16:05:36.564692 5750 leader_election.cc:302] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97c89026d44e402093c6635a028db884; no voters:
I20260924 16:05:36.565392 5750 leader_election.cc:288] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:05:36.565574 5755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:36.566473 5755 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884 [term 1 LEADER]: Becoming Leader. State: Replica: 97c89026d44e402093c6635a028db884, State: Running, Role: LEADER
I20260924 16:05:36.567230 5755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c89026d44e402093c6635a028db884" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43765 } }
I20260924 16:05:36.568243 5750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:05:36.570989 5757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97c89026d44e402093c6635a028db884. Latest consensus state: current_term: 1 leader_uuid: "97c89026d44e402093c6635a028db884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c89026d44e402093c6635a028db884" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43765 } } }
I20260924 16:05:36.571193 5756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97c89026d44e402093c6635a028db884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c89026d44e402093c6635a028db884" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43765 } } }
I20260924 16:05:36.571592 5757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:36.571604 5756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:36.573702 5764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:05:36.581189 5764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:05:36.598155 5764 catalog_manager.cc:1347] Generated new cluster ID: 3a24a125f77d42af8b249c127c2a542e
I20260924 16:05:36.598361 5764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:05:36.633841 5764 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:05:36.635885 5764 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:05:36.649986 5764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884: Generated new TSK 0
I20260924 16:05:36.651487 5764 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.214.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
I20260924 16:05:36.722540 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:43765
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:37.092814 5776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:37.093146 5776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:37.093259 5776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:37.101125 5776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:37.101269 5776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:37.101359 5776 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 16:05:37.101418 5776 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:05:37.101493 5776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:05:37.112376 5776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.214.254:43765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:37.113937 5776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:37.115428 5776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:37.129891 5781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:37.129855 5782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:37.130532 5784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:37.131199 5776 server_base.cc:1034] running on GCE node
I20260924 16:05:37.132026 5776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:37.133309 5776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:37.134565 5776 hybrid_clock.cc:648] HybridClock initialized: now 1790265937134484 us; error 56 us; skew 500 ppm
I20260924 16:05:37.135003 5776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:37 dist-test-slave-1gc3 krb5kdc[5658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265937, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:37.143262 5776 init.cc:377] Logged in from keytab as kudu/127.3.214.193@KRBTEST.COM (short username kudu)
I20260924 16:05:37.145341 5776 webserver.cc:460] Webserver started at http://127.3.214.193:41393/ using document root <none> and password file <none>
I20260924 16:05:37.146353 5776 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:37.146515 5776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:37.146896 5776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:37.150904 5776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "1e09099dc5684c09bcb34725ff82bc94"
format_stamp: "Formatted at 2026-09-24 16:05:37 on dist-test-slave-1gc3"
server_key: "ff93dc2668646a0bff45d3c22271bc84"
server_key_iv: "d72c265edd43b8b3d1ecf4b2f57bca91"
server_key_version: "encryptionkey@0"
I20260924 16:05:37.151820 5776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "1e09099dc5684c09bcb34725ff82bc94"
format_stamp: "Formatted at 2026-09-24 16:05:37 on dist-test-slave-1gc3"
server_key: "ff93dc2668646a0bff45d3c22271bc84"
server_key_iv: "d72c265edd43b8b3d1ecf4b2f57bca91"
server_key_version: "encryptionkey@0"
I20260924 16:05:37.158586 5776 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260924 16:05:37.163242 5792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:37.165163 5776 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:05:37.165385 5776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "1e09099dc5684c09bcb34725ff82bc94"
format_stamp: "Formatted at 2026-09-24 16:05:37 on dist-test-slave-1gc3"
server_key: "ff93dc2668646a0bff45d3c22271bc84"
server_key_iv: "d72c265edd43b8b3d1ecf4b2f57bca91"
server_key_version: "encryptionkey@0"
I20260924 16:05:37.165675 5776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:37.192111 5776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:37.196861 5776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:37.197188 5776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:37.200524 5776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:37.200650 5776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:37.200778 5776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:37.200865 5776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:05:37 dist-test-slave-1gc3 krb5kdc[5658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265937, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:37.230505 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:37.201628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:40453 (local address 127.3.214.254:43765)
0924 16:05:37.202273 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:37.202282 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:37.203960 (+ 1678us) server_negotiation.cc:407] Connection header received
0924 16:05:37.207415 (+ 3455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:37.207424 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:37.207579 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:37.207753 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:37.210667 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:37.211881 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:37.214637 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:37.215315 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:37.215503 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:37.221620 (+ 6117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:37.221629 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:37.221634 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:37.221691 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:37.225444 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:37.226501 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:37.226508 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:37.226511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:37.226698 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:37.227687 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:37.227692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:37.227694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:37.228065 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:37.228242 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:37.229799 (+ 1557us) server_negotiation.cc:299] Negotiation successful
0924 16:05:37.230040 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":280,"threads_started":1}
I20260924 16:05:37.232874 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:37.202551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43765 (local address 127.3.214.193:40453)
0924 16:05:37.203695 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:37.203778 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:37.206949 (+ 3171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:37.208181 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:37.208200 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:37.209161 (+ 961us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:37.210336 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:37.210366 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:37.212069 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:37.212073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:37.214425 (+ 2352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:37.214446 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:37.215672 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:37.215682 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:37.215954 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:37.217060 (+ 1106us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:37.217107 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:37.221183 (+ 4076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:37.225646 (+ 4463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:37.225659 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:37.225681 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:37.226340 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:37.226906 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:37.226915 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:37.226920 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:37.227412 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:37.228219 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:37.228231 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:37.229096 (+ 865us) client_negotiation.cc:769] Sending connection context
0924 16:05:37.229646 (+ 550us) client_negotiation.cc:240] Negotiation successful
0924 16:05:37.230504 (+ 858us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":413,"threads_started":1}
I20260924 16:05:37.256349 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:37.241311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43765 (local address 127.3.214.193:45073)
0924 16:05:37.241996 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:37.242015 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:37.242215 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:37.242566 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:37.242571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:37.243140 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:37.243488 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:37.243499 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:37.244597 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:37.244601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:37.245781 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:37.245798 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:37.247181 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:37.247187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:37.247339 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:37.248177 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:37.248200 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:37.249443 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:37.253334 (+ 3891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:37.253344 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:37.253349 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:37.253988 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:37.254434 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:37.254439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:37.254443 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:37.254610 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:37.255196 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:37.255202 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:37.255436 (+ 234us) client_negotiation.cc:769] Sending connection context
0924 16:05:37.255594 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 16:05:37.255950 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":373,"threads_started":1}
I20260924 16:05:37.257220 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:37.241282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:45073 (local address 127.3.214.254:43765)
0924 16:05:37.241466 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:37.241470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:37.242139 (+ 669us) server_negotiation.cc:407] Connection header received
0924 16:05:37.242350 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:37.242354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:37.242432 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:37.242518 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:37.243632 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:37.244463 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:37.246079 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:37.246918 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:37.247133 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:37.249661 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:37.249668 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:37.249674 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:37.249723 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:37.253148 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:37.254143 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:37.254149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:37.254153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:37.254299 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:37.254723 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:37.254729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:37.254732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:37.255054 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:37.255201 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:37.256721 (+ 1520us) server_negotiation.cc:299] Negotiation successful
0924 16:05:37.256949 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:05:37.259404 5776 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:36971
I20260924 16:05:37.259419 5914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:36971 every 8 connection(s)
I20260924 16:05:37.261197 5776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:05:37.263801 5915 heartbeater.cc:347] Connected to a master server at 127.3.214.254:43765
I20260924 16:05:37.264056 5915 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:37.264794 5915 heartbeater.cc:511] Master 127.3.214.254:43765 requested a full tablet report, sending...
I20260924 16:05:37.266283 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 5776
I20260924 16:05:37.266503 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:05:37.267225 3931 external_mini_cluster.cc:1428] Setting key d5b9f60c424e4021d56ff9e8085b96ae
I20260924 16:05:37.267194 5703 ts_manager.cc:194] Registered new tserver with Master: 1e09099dc5684c09bcb34725ff82bc94 (127.3.214.193:36971)
I20260924 16:05:37.269150 5703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:40453
WARNING: no policy specified for kudu/127.3.214.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
I20260924 16:05:37.313452 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:43765
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:37.655879 5919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:37.656188 5919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:37.656294 5919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:37.663993 5919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:37.664134 5919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:37.664239 5919 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 16:05:37.664296 5919 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:05:37.664361 5919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:05:37.675014 5919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.214.254:43765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:37.676347 5919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:37.677624 5919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:37.692425 5926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:37.692406 5925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:37.692692 5928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:37.693799 5919 server_base.cc:1034] running on GCE node
I20260924 16:05:37.694633 5919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:37.695747 5919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:37.697008 5919 hybrid_clock.cc:648] HybridClock initialized: now 1790265937696933 us; error 62 us; skew 500 ppm
I20260924 16:05:37.697489 5919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:37 dist-test-slave-1gc3 krb5kdc[5658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265937, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:37.701107 5919 init.cc:377] Logged in from keytab as kudu/127.3.214.194@KRBTEST.COM (short username kudu)
I20260924 16:05:37.702975 5919 webserver.cc:460] Webserver started at http://127.3.214.194:35489/ using document root <none> and password file <none>
I20260924 16:05:37.703763 5919 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:37.703890 5919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:37.704228 5919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:37.708269 5919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "b57fe08e069c4e84a45f86322e3ed026"
format_stamp: "Formatted at 2026-09-24 16:05:37 on dist-test-slave-1gc3"
server_key: "0e478e0c10d39525574c8d28147d884b"
server_key_iv: "e9256e6ccc1f2ddfe8fa215ffd2001ac"
server_key_version: "encryptionkey@0"
I20260924 16:05:37.709369 5919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "b57fe08e069c4e84a45f86322e3ed026"
format_stamp: "Formatted at 2026-09-24 16:05:37 on dist-test-slave-1gc3"
server_key: "0e478e0c10d39525574c8d28147d884b"
server_key_iv: "e9256e6ccc1f2ddfe8fa215ffd2001ac"
server_key_version: "encryptionkey@0"
I20260924 16:05:37.716338 5919 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.007s
I20260924 16:05:37.720891 5935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:37.722784 5919 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:05:37.723001 5919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "b57fe08e069c4e84a45f86322e3ed026"
format_stamp: "Formatted at 2026-09-24 16:05:37 on dist-test-slave-1gc3"
server_key: "0e478e0c10d39525574c8d28147d884b"
server_key_iv: "e9256e6ccc1f2ddfe8fa215ffd2001ac"
server_key_version: "encryptionkey@0"
I20260924 16:05:37.723230 5919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:37.761114 5919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:37.766422 5919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:37.766750 5919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:37.770409 5919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:37.770547 5919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:37.770689 5919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:37.770781 5919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:05:37 dist-test-slave-1gc3 krb5kdc[5658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265937, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:37.801024 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:37.771535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:56379 (local address 127.3.214.254:43765)
0924 16:05:37.772125 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:37.772133 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:37.774509 (+ 2376us) server_negotiation.cc:407] Connection header received
0924 16:05:37.777914 (+ 3405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:37.777923 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:37.778059 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:37.778200 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:37.781099 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:37.782448 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:37.785257 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:37.785856 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:37.785998 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:37.792715 (+ 6717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:37.792722 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:37.792726 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:37.792775 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:37.796199 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:37.797428 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:37.797434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:37.797437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:37.797604 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:37.798477 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:37.798481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:37.798483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:37.798919 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:37.799108 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:37.800262 (+ 1154us) server_negotiation.cc:299] Negotiation successful
0924 16:05:37.800580 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":1896,"threads_started":1}
I20260924 16:05:37.802600 5945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:37.773005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43765 (local address 127.3.214.194:56379)
0924 16:05:37.774314 (+ 1309us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:37.774369 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:37.776665 (+ 2296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:37.778865 (+ 2200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:37.778879 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:37.779797 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:37.780762 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:37.780794 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:37.782730 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:37.782735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:37.784930 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:37.784951 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:37.786407 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:37.786418 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:37.787001 (+ 583us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:37.788130 (+ 1129us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:37.788168 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:37.792234 (+ 4066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:37.796480 (+ 4246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:37.796494 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:37.796517 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:37.797282 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:37.797762 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:37.797768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:37.797772 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:37.798311 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:37.799127 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:37.799139 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:37.799722 (+ 583us) client_negotiation.cc:769] Sending connection context
0924 16:05:37.800122 (+ 400us) client_negotiation.cc:240] Negotiation successful
0924 16:05:37.800793 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"thread_start_us":614,"threads_started":1}
I20260924 16:05:37.828212 5919 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:38993
I20260924 16:05:37.828212 6058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:38993 every 8 connection(s)
I20260924 16:05:37.829396 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:37.813267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:51535 (local address 127.3.214.254:43765)
0924 16:05:37.813553 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:37.813562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:37.814766 (+ 1204us) server_negotiation.cc:407] Connection header received
0924 16:05:37.814957 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:37.814964 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:37.815079 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:37.815249 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:37.816246 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:37.817441 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:37.819488 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:37.820113 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:37.820257 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:37.822814 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:37.822821 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:37.822826 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:37.822872 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:37.826193 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:37.827008 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:37.827013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:37.827016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:37.827183 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:37.827632 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:37.827637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:37.827641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:37.828129 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:37.828276 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:37.828770 (+ 494us) server_negotiation.cc:299] Negotiation successful
0924 16:05:37.829025 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260924 16:05:37.829566 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:37.813110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43765 (local address 127.3.214.194:51535)
0924 16:05:37.814617 (+ 1507us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:37.814643 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:37.814803 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:37.815289 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:37.815293 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:37.815755 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:37.816080 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:37.816089 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:37.817661 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:37.817665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:37.819238 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:37.819264 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:37.820334 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:37.820342 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:37.820525 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:37.821211 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:37.821238 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:37.822566 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:37.826352 (+ 3786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:37.826375 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:37.826379 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:37.826878 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:37.827323 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:37.827328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:37.827331 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:37.827505 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:37.828246 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:37.828256 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:37.828473 (+ 217us) client_negotiation.cc:769] Sending connection context
0924 16:05:37.828623 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 16:05:37.829114 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1337,"thread_start_us":1177,"threads_started":1}
I20260924 16:05:37.830255 5919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:05:37.833158 6060 heartbeater.cc:347] Connected to a master server at 127.3.214.254:43765
I20260924 16:05:37.833485 6060 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:37.834456 6060 heartbeater.cc:511] Master 127.3.214.254:43765 requested a full tablet report, sending...
I20260924 16:05:37.836337 5703 ts_manager.cc:194] Registered new tserver with Master: b57fe08e069c4e84a45f86322e3ed026 (127.3.214.194:38993)
I20260924 16:05:37.837376 5703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:56379
I20260924 16:05:37.837387 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 5919
I20260924 16:05:37.837570 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:05:37.838037 3931 external_mini_cluster.cc:1428] Setting key 246da4263af9bf0f7d66a7023e57a261
WARNING: no policy specified for kudu/127.3.214.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
I20260924 16:05:37.886709 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:43765
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:05:38.272837 5915 heartbeater.cc:503] Master 127.3.214.254:43765 was elected leader, sending a full tablet report...
W20260924 16:05:38.290004 6064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:38.290405 6064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:38.290551 6064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:38.298962 6064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:38.299170 6064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:38.299366 6064 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 16:05:38.299448 6064 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:05:38.299576 6064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:05:38.312016 6064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.214.254:43765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:38.314008 6064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:38.315830 6064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:38.331996 6069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:38.331964 6070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:38.332044 6072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:38.333088 6064 server_base.cc:1034] running on GCE node
I20260924 16:05:38.333889 6064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:38.335201 6064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:38.336503 6064 hybrid_clock.cc:648] HybridClock initialized: now 1790265938336383 us; error 71 us; skew 500 ppm
I20260924 16:05:38.336947 6064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:38 dist-test-slave-1gc3 krb5kdc[5658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265938, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:38.340667 6064 init.cc:377] Logged in from keytab as kudu/127.3.214.195@KRBTEST.COM (short username kudu)
I20260924 16:05:38.342588 6064 webserver.cc:460] Webserver started at http://127.3.214.195:40729/ using document root <none> and password file <none>
I20260924 16:05:38.343393 6064 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:38.343535 6064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:38.343888 6064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:38.347396 6064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "2b7eae966d1c4f11b8c6f273ea61c46b"
format_stamp: "Formatted at 2026-09-24 16:05:38 on dist-test-slave-1gc3"
server_key: "1faa9f112da5ef665011e1ebc1c856db"
server_key_iv: "cd58c7c436cfd04e9ae893112060bfaf"
server_key_version: "encryptionkey@0"
I20260924 16:05:38.348296 6064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "2b7eae966d1c4f11b8c6f273ea61c46b"
format_stamp: "Formatted at 2026-09-24 16:05:38 on dist-test-slave-1gc3"
server_key: "1faa9f112da5ef665011e1ebc1c856db"
server_key_iv: "cd58c7c436cfd04e9ae893112060bfaf"
server_key_version: "encryptionkey@0"
I20260924 16:05:38.354923 6064 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260924 16:05:38.359606 6079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:38.361548 6064 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260924 16:05:38.361768 6064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "2b7eae966d1c4f11b8c6f273ea61c46b"
format_stamp: "Formatted at 2026-09-24 16:05:38 on dist-test-slave-1gc3"
server_key: "1faa9f112da5ef665011e1ebc1c856db"
server_key_iv: "cd58c7c436cfd04e9ae893112060bfaf"
server_key_version: "encryptionkey@0"
I20260924 16:05:38.362020 6064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:38.377146 6064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:38.381632 6064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:38.381888 6064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:38.385205 6064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:38.385329 6064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:38.385453 6064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:38.385545 6064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:05:38 dist-test-slave-1gc3 krb5kdc[5658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265938, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:38.416000 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:38.386293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:42397 (local address 127.3.214.254:43765)
0924 16:05:38.386872 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:38.386878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:38.388182 (+ 1304us) server_negotiation.cc:407] Connection header received
0924 16:05:38.391344 (+ 3162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:38.391350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:38.391463 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:38.391580 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:38.394736 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.395950 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.399270 (+ 3320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.399939 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.400107 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:38.406461 (+ 6354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:38.406469 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:38.406476 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:38.406544 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:38.410902 (+ 4358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:38.412472 (+ 1570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:38.412479 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:38.412483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:38.412714 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:38.413365 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:38.413371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:38.413375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:38.413752 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:38.413923 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:38.415169 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0924 16:05:38.415495 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":304,"threads_started":1}
I20260924 16:05:38.417439 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:38.386903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43765 (local address 127.3.214.195:42397)
0924 16:05:38.387957 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:38.388018 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:38.390873 (+ 2855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:38.391936 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:38.391946 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:38.393055 (+ 1109us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:38.394323 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:38.394368 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.396286 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.396293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:38.398965 (+ 2672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:38.399036 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.400418 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.400431 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:38.400871 (+ 440us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:38.401832 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:38.401873 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:38.405922 (+ 4049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:38.411194 (+ 5272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:38.411212 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:38.411238 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:38.412229 (+ 991us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:38.412873 (+ 644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:38.412885 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:38.412891 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:38.413217 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:38.413924 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:38.413935 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:38.414598 (+ 663us) client_negotiation.cc:769] Sending connection context
0924 16:05:38.415039 (+ 441us) client_negotiation.cc:240] Negotiation successful
0924 16:05:38.415849 (+ 810us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":444,"threads_started":1}
I20260924 16:05:38.441601 6064 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:46149
I20260924 16:05:38.441641 6201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:46149 every 8 connection(s)
I20260924 16:05:38.443423 6064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:05:38.446446 6202 heartbeater.cc:347] Connected to a master server at 127.3.214.254:43765
I20260924 16:05:38.446772 6202 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:38.447777 6202 heartbeater.cc:511] Master 127.3.214.254:43765 requested a full tablet report, sending...
I20260924 16:05:38.447871 6175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:38.430095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43765 (local address 127.3.214.195:42321)
0924 16:05:38.431101 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:38.431127 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:38.431330 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:38.431731 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:38.431736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:38.432470 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:38.433208 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:38.433226 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.434979 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.434986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:38.436663 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:38.436689 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.437830 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.437838 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:38.438056 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:38.438986 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:38.439015 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:38.440106 (+ 1091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:38.444111 (+ 4005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:38.444118 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:38.444142 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:38.444918 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:38.445524 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:38.445531 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:38.445535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:38.445742 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:38.446645 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:38.446651 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:38.446936 (+ 285us) client_negotiation.cc:769] Sending connection context
0924 16:05:38.447094 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 16:05:38.447450 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"thread_start_us":593,"threads_started":1}
I20260924 16:05:38.447897 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:38.429988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:42321 (local address 127.3.214.254:43765)
0924 16:05:38.430255 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:38.430262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:38.431249 (+ 987us) server_negotiation.cc:407] Connection header received
0924 16:05:38.431478 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:38.431484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:38.431571 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:38.431662 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:38.433497 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.434765 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.436860 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.437548 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.437705 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:38.440368 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:38.440375 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:38.440380 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:38.440424 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:38.443914 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:38.445119 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:38.445126 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:38.445131 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:38.445332 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:38.445972 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:38.445977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:38.445980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:38.446449 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:38.446641 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:38.447243 (+ 602us) server_negotiation.cc:299] Negotiation successful
0924 16:05:38.447551 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 16:05:38.449636 5702 ts_manager.cc:194] Registered new tserver with Master: 2b7eae966d1c4f11b8c6f273ea61c46b (127.3.214.195:46149)
I20260924 16:05:38.450706 5702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:42397
I20260924 16:05:38.451295 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 6064
I20260924 16:05:38.451498 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:05:38.452037 3931 external_mini_cluster.cc:1428] Setting key 3580b53b078fc54c7a3bcbc1ebe27cf1
I20260924 16:05:38.455498 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 16:05:38 dist-test-slave-1gc3 krb5kdc[5658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 16:05:38 dist-test-slave-1gc3 krb5kdc[5658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:38.509140 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:38.494318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43854 (local address 127.3.214.254:43765)
0924 16:05:38.494610 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:38.494618 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:38.494822 (+ 204us) server_negotiation.cc:407] Connection header received
0924 16:05:38.495071 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:38.495076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:38.495157 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:38.495248 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:38.496726 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.497452 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.498739 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.499142 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.499251 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:38.503553 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:38.503558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:38.503562 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:38.503603 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:38.506499 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:38.507214 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:38.507218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:38.507220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:38.507358 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:38.507727 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:38.507730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:38.507732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:38.508029 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:38.508282 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:38.508582 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 16:05:38.508804 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260924 16:05:38.514142 5702 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:05:38.521574 5702 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:05:38.562489 6215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:38.537519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:46149 (local address 127.0.0.1:51890)
0924 16:05:38.551228 (+ 13709us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:38.551255 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:38.551415 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:38.554279 (+ 2864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:38.554283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:38.554334 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:38.554821 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:38.554841 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.556604 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.556610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:38.559132 (+ 2522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:38.559152 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.561073 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.561080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:38.561356 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:38.561432 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 16:05:38.561625 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 16:05:38.561905 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13501,"mutex_wait_us":21,"spinlock_wait_cycles":25984,"thread_start_us":11418,"threads_started":1}
I20260924 16:05:38.563472 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:38.537907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51890 (local address 127.3.214.195:46149)
0924 16:05:38.553255 (+ 15348us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:38.553266 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:38.553313 (+ 47us) server_negotiation.cc:407] Connection header received
0924 16:05:38.553458 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:38.553466 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:38.553773 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:38.554625 (+ 852us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:38.555214 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.556449 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.559353 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.560813 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.560979 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:38.561242 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:38.562823 (+ 1581us) server_negotiation.cc:299] Negotiation successful
0924 16:05:38.563078 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15110,"thread_start_us":16034,"threads_started":1}
I20260924 16:05:38.563526 6217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:38.537460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:38993 (local address 127.0.0.1:41540)
0924 16:05:38.549560 (+ 12100us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:38.549644 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:38.549936 (+ 292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:38.554247 (+ 4311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:38.554255 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:38.554307 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:38.555351 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:38.555367 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.558453 (+ 3086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.558459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:38.560402 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:38.560416 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.562717 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.562723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:38.562883 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:38.562915 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 16:05:38.563004 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 16:05:38.563142 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11670,"mutex_wait_us":6,"thread_start_us":13359,"threads_started":1}
I20260924 16:05:38.564432 6216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:38.537608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41540 (local address 127.3.214.194:38993)
0924 16:05:38.553552 (+ 15944us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:38.553560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:38.553585 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:05:38.553665 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:38.553671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:38.553848 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:38.554038 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:38.556794 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.558259 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.560611 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.562150 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.562305 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:38.562537 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:38.563921 (+ 1384us) server_negotiation.cc:299] Negotiation successful
0924 16:05:38.564135 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15769,"thread_start_us":16755,"threads_started":1}
I20260924 16:05:38.564787 6219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:38.551474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:36971 (local address 127.0.0.1:42028)
0924 16:05:38.555360 (+ 3886us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:38.555387 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:38.555550 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:38.556779 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:38.556784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:38.556812 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:38.557354 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:38.557372 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.559740 (+ 2368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.559746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:38.561765 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:38.561781 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.563894 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.563919 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:38.564102 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:38.564131 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 16:05:38.564219 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 16:05:38.564368 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3694,"mutex_wait_us":2,"thread_start_us":3440,"threads_started":1}
I20260924 16:05:38.564764 6220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:38.551836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42028 (local address 127.3.214.193:36971)
0924 16:05:38.556126 (+ 4290us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:38.556134 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:38.556171 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:05:38.556273 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:38.556279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:38.556542 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:38.557547 (+ 1005us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:38.557833 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.559498 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.562018 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.563652 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.563795 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:38.564027 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:38.564288 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 16:05:38.564472 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4119,"thread_start_us":3958,"threads_started":1}
I20260924 16:05:38.568785 6131 tablet_service.cc:1467] Processing CreateTablet for tablet 0303b8e6468b49adaf30881c626f6b28 (DEFAULT_TABLE table=test-table [id=941a43c0704d47d3814790d287699093]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:38.568785 5986 tablet_service.cc:1467] Processing CreateTablet for tablet 0303b8e6468b49adaf30881c626f6b28 (DEFAULT_TABLE table=test-table [id=941a43c0704d47d3814790d287699093]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:38.569068 5844 tablet_service.cc:1467] Processing CreateTablet for tablet 0303b8e6468b49adaf30881c626f6b28 (DEFAULT_TABLE table=test-table [id=941a43c0704d47d3814790d287699093]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:38.570565 5844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0303b8e6468b49adaf30881c626f6b28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:38.570606 5986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0303b8e6468b49adaf30881c626f6b28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:38.570607 6131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0303b8e6468b49adaf30881c626f6b28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:38.582883 6221 tablet_bootstrap.cc:492] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94: Bootstrap starting.
I20260924 16:05:38.586565 6221 tablet_bootstrap.cc:654] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:38.587666 6223 tablet_bootstrap.cc:492] T 0303b8e6468b49adaf30881c626f6b28 P 2b7eae966d1c4f11b8c6f273ea61c46b: Bootstrap starting.
I20260924 16:05:38.587980 6221 log.cc:824] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:38.591059 6222 tablet_bootstrap.cc:492] T 0303b8e6468b49adaf30881c626f6b28 P b57fe08e069c4e84a45f86322e3ed026: Bootstrap starting.
I20260924 16:05:38.592238 6223 tablet_bootstrap.cc:654] T 0303b8e6468b49adaf30881c626f6b28 P 2b7eae966d1c4f11b8c6f273ea61c46b: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:38.593622 6223 log.cc:824] T 0303b8e6468b49adaf30881c626f6b28 P 2b7eae966d1c4f11b8c6f273ea61c46b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:38.594856 6222 tablet_bootstrap.cc:654] T 0303b8e6468b49adaf30881c626f6b28 P b57fe08e069c4e84a45f86322e3ed026: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:38.596256 6222 log.cc:824] T 0303b8e6468b49adaf30881c626f6b28 P b57fe08e069c4e84a45f86322e3ed026: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:38.600106 6222 tablet_bootstrap.cc:492] T 0303b8e6468b49adaf30881c626f6b28 P b57fe08e069c4e84a45f86322e3ed026: No bootstrap required, opened a new log
I20260924 16:05:38.600106 6221 tablet_bootstrap.cc:492] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94: No bootstrap required, opened a new log
I20260924 16:05:38.600176 6223 tablet_bootstrap.cc:492] T 0303b8e6468b49adaf30881c626f6b28 P 2b7eae966d1c4f11b8c6f273ea61c46b: No bootstrap required, opened a new log
I20260924 16:05:38.600461 6221 ts_tablet_manager.cc:1397] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.003s
I20260924 16:05:38.600446 6222 ts_tablet_manager.cc:1397] T 0303b8e6468b49adaf30881c626f6b28 P b57fe08e069c4e84a45f86322e3ed026: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260924 16:05:38.600586 6223 ts_tablet_manager.cc:1397] T 0303b8e6468b49adaf30881c626f6b28 P 2b7eae966d1c4f11b8c6f273ea61c46b: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260924 16:05:38.610991 6223 raft_consensus.cc:348] T 0303b8e6468b49adaf30881c626f6b28 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } } peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } }
I20260924 16:05:38.611552 6223 raft_consensus.cc:374] T 0303b8e6468b49adaf30881c626f6b28 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:38.611688 6223 raft_consensus.cc:729] T 0303b8e6468b49adaf30881c626f6b28 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b7eae966d1c4f11b8c6f273ea61c46b, State: Initialized, Role: FOLLOWER
I20260924 16:05:38.611536 6221 raft_consensus.cc:348] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } } peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } }
I20260924 16:05:38.611536 6222 raft_consensus.cc:348] T 0303b8e6468b49adaf30881c626f6b28 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } } peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } }
I20260924 16:05:38.612084 6221 raft_consensus.cc:374] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:38.612084 6222 raft_consensus.cc:374] T 0303b8e6468b49adaf30881c626f6b28 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:38.612198 6221 raft_consensus.cc:729] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e09099dc5684c09bcb34725ff82bc94, State: Initialized, Role: FOLLOWER
I20260924 16:05:38.612198 6222 raft_consensus.cc:729] T 0303b8e6468b49adaf30881c626f6b28 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57fe08e069c4e84a45f86322e3ed026, State: Initialized, Role: FOLLOWER
I20260924 16:05:38.612696 6223 consensus_queue.cc:260] T 0303b8e6468b49adaf30881c626f6b28 P 2b7eae966d1c4f11b8c6f273ea61c46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } } peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } }
I20260924 16:05:38.612960 6222 consensus_queue.cc:260] T 0303b8e6468b49adaf30881c626f6b28 P b57fe08e069c4e84a45f86322e3ed026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } } peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } }
I20260924 16:05:38.612952 6221 consensus_queue.cc:260] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } } peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } }
I20260924 16:05:38.614651 6060 heartbeater.cc:503] Master 127.3.214.254:43765 was elected leader, sending a full tablet report...
I20260924 16:05:38.615265 6222 ts_tablet_manager.cc:1428] T 0303b8e6468b49adaf30881c626f6b28 P b57fe08e069c4e84a45f86322e3ed026: Time spent starting tablet: real 0.015s user 0.013s sys 0.004s
I20260924 16:05:38.615655 6221 ts_tablet_manager.cc:1428] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94: Time spent starting tablet: real 0.015s user 0.013s sys 0.003s
I20260924 16:05:38.617215 6202 heartbeater.cc:503] Master 127.3.214.254:43765 was elected leader, sending a full tablet report...
I20260924 16:05:38.617859 6223 ts_tablet_manager.cc:1428] T 0303b8e6468b49adaf30881c626f6b28 P 2b7eae966d1c4f11b8c6f273ea61c46b: Time spent starting tablet: real 0.017s user 0.010s sys 0.007s
I20260924 16:05:38.637817 6229 raft_consensus.cc:482] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:05:38.638319 6229 raft_consensus.cc:504] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } } peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } }
I20260924 16:05:38.640656 6229 leader_election.cc:288] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b57fe08e069c4e84a45f86322e3ed026 (127.3.214.194:38993), 2b7eae966d1c4f11b8c6f273ea61c46b (127.3.214.195:46149)
I20260924 16:05:38.648506 6216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:38.641107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:54715 (local address 127.3.214.194:38993)
0924 16:05:38.641326 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:38.641333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:38.642354 (+ 1021us) server_negotiation.cc:407] Connection header received
0924 16:05:38.642570 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:38.642575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:38.642655 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:38.642760 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:38.643570 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.644297 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.646532 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.647369 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.647529 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:38.647611 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:38.648086 (+ 475us) server_negotiation.cc:299] Negotiation successful
0924 16:05:38.648217 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 16:05:38.648640 6230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:38.641074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:46149 (local address 127.3.214.193:55581)
0924 16:05:38.641926 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:38.641950 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:38.642170 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:38.642604 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:38.642608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:38.642637 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:38.643174 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:38.643186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.644400 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.644405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:38.646221 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:38.646234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.647713 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.647718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:38.647850 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:38.647913 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 16:05:38.647996 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 16:05:38.648148 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":85,"thread_start_us":428,"threads_started":1}
I20260924 16:05:38.648633 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:38.641265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:55581 (local address 127.3.214.195:46149)
0924 16:05:38.641531 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:38.641540 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:38.642077 (+ 537us) server_negotiation.cc:407] Connection header received
0924 16:05:38.642354 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:38.642361 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:38.642456 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:38.642564 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:38.643353 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.644252 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.646414 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.647368 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.647566 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:38.647675 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:38.648069 (+ 394us) server_negotiation.cc:299] Negotiation successful
0924 16:05:38.648225 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 16:05:38.649250 6231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:38.641109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:38993 (local address 127.3.214.193:54715)
0924 16:05:38.642221 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:38.642242 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:38.642419 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:38.642812 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:38.642815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:38.642843 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:38.643388 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:38.643402 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.644453 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.644463 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:38.646321 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:38.646339 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.647658 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.647664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:38.647835 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:38.647922 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 16:05:38.648034 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 16:05:38.648223 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":77,"thread_start_us":695,"threads_started":1}
I20260924 16:05:38.649838 6151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0303b8e6468b49adaf30881c626f6b28" candidate_uuid: "1e09099dc5684c09bcb34725ff82bc94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" is_pre_election: true
I20260924 16:05:38.650370 6008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0303b8e6468b49adaf30881c626f6b28" candidate_uuid: "1e09099dc5684c09bcb34725ff82bc94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57fe08e069c4e84a45f86322e3ed026" is_pre_election: true
I20260924 16:05:38.650535 6151 raft_consensus.cc:2454] T 0303b8e6468b49adaf30881c626f6b28 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e09099dc5684c09bcb34725ff82bc94 in term 0.
I20260924 16:05:38.650882 6008 raft_consensus.cc:2454] T 0303b8e6468b49adaf30881c626f6b28 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e09099dc5684c09bcb34725ff82bc94 in term 0.
I20260924 16:05:38.651564 5796 leader_election.cc:302] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e09099dc5684c09bcb34725ff82bc94, 2b7eae966d1c4f11b8c6f273ea61c46b; no voters:
I20260924 16:05:38.652094 6229 raft_consensus.cc:2781] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:05:38.652285 6229 raft_consensus.cc:482] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:05:38.652408 6229 raft_consensus.cc:3037] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:38.655184 6229 raft_consensus.cc:504] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } } peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } }
I20260924 16:05:38.655964 6229 leader_election.cc:288] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94 [CANDIDATE]: Term 1 election: Requested vote from peers b57fe08e069c4e84a45f86322e3ed026 (127.3.214.194:38993), 2b7eae966d1c4f11b8c6f273ea61c46b (127.3.214.195:46149)
I20260924 16:05:38.656615 6008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0303b8e6468b49adaf30881c626f6b28" candidate_uuid: "1e09099dc5684c09bcb34725ff82bc94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57fe08e069c4e84a45f86322e3ed026"
I20260924 16:05:38.656692 6151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0303b8e6468b49adaf30881c626f6b28" candidate_uuid: "1e09099dc5684c09bcb34725ff82bc94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b"
I20260924 16:05:38.656848 6008 raft_consensus.cc:3037] T 0303b8e6468b49adaf30881c626f6b28 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:38.656967 6151 raft_consensus.cc:3037] T 0303b8e6468b49adaf30881c626f6b28 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:38.660023 6008 raft_consensus.cc:2454] T 0303b8e6468b49adaf30881c626f6b28 P b57fe08e069c4e84a45f86322e3ed026 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e09099dc5684c09bcb34725ff82bc94 in term 1.
I20260924 16:05:38.660346 6151 raft_consensus.cc:2454] T 0303b8e6468b49adaf30881c626f6b28 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e09099dc5684c09bcb34725ff82bc94 in term 1.
I20260924 16:05:38.660558 5793 leader_election.cc:302] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e09099dc5684c09bcb34725ff82bc94, b57fe08e069c4e84a45f86322e3ed026; no voters:
I20260924 16:05:38.660962 6229 raft_consensus.cc:2781] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:38.661490 6229 raft_consensus.cc:686] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94 [term 1 LEADER]: Becoming Leader. State: Replica: 1e09099dc5684c09bcb34725ff82bc94, State: Running, Role: LEADER
I20260924 16:05:38.662194 6229 consensus_queue.cc:237] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } } peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } }
I20260924 16:05:38.668421 5703 catalog_manager.cc:5495] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e09099dc5684c09bcb34725ff82bc94 (127.3.214.193). New cstate: current_term: 1 leader_uuid: "1e09099dc5684c09bcb34725ff82bc94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } health_report { overall_health: HEALTHY } } }
W20260924 16:05:38.696394 6203 tablet.cc:2369] T 0303b8e6468b49adaf30881c626f6b28 P 2b7eae966d1c4f11b8c6f273ea61c46b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 16:05:38 dist-test-slave-1gc3 krb5kdc[5658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265936, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:38.738677 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:38.721611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43765 (local address 127.0.0.1:43856)
0924 16:05:38.722474 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:38.722496 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:38.722679 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:38.723166 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:38.723170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:38.724095 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:38.724480 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:38.724491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.725670 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.725676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:38.727178 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:38.727196 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.728261 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.728268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:38.728417 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:38.729242 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:38.729267 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:38.731953 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:38.735247 (+ 3294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:38.735267 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:38.735284 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:38.735997 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:38.736537 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:38.736541 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:38.736544 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:38.736761 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:38.737437 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:38.737446 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:38.737658 (+ 212us) client_negotiation.cc:769] Sending connection context
0924 16:05:38.737885 (+ 227us) client_negotiation.cc:240] Negotiation successful
0924 16:05:38.738269 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":427,"threads_started":1}
I20260924 16:05:38.738981 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:38.721536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43856 (local address 127.3.214.254:43765)
0924 16:05:38.721779 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:38.721787 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:38.722652 (+ 865us) server_negotiation.cc:407] Connection header received
0924 16:05:38.722857 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:38.722862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:38.722958 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:38.723066 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:38.724665 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.725478 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.727356 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.728018 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:38.728132 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:38.732224 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:38.732229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:38.732233 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:38.732277 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:38.735039 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:38.736202 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:38.736208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:38.736211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:38.736410 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:38.736945 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:38.736951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:38.736954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:38.737296 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:38.737436 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:38.738067 (+ 631us) server_negotiation.cc:299] Negotiation successful
0924 16:05:38.738516 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":89}
I20260924 16:05:38.745553 5703 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:43856:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 16:05:38.746869 5703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:05:38.755549 5844 tablet_service.cc:1467] Processing CreateTablet for tablet 85bcefd82e5b46d5977700be540f2621 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7cf7002d2b014a8f81a6db2b77e02b9d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:05:38.755797 5986 tablet_service.cc:1467] Processing CreateTablet for tablet 85bcefd82e5b46d5977700be540f2621 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7cf7002d2b014a8f81a6db2b77e02b9d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:05:38.756122 6131 tablet_service.cc:1467] Processing CreateTablet for tablet 85bcefd82e5b46d5977700be540f2621 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7cf7002d2b014a8f81a6db2b77e02b9d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:05:38.756773 5844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85bcefd82e5b46d5977700be540f2621. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:38.756994 5986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85bcefd82e5b46d5977700be540f2621. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:38.757141 6131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85bcefd82e5b46d5977700be540f2621. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:38.764427 6222 tablet_bootstrap.cc:492] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026: Bootstrap starting.
W20260924 16:05:38.764407 5916 tablet.cc:2369] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:05:38.764747 6223 tablet_bootstrap.cc:492] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b: Bootstrap starting.
I20260924 16:05:38.765519 6221 tablet_bootstrap.cc:492] T 85bcefd82e5b46d5977700be540f2621 P 1e09099dc5684c09bcb34725ff82bc94: Bootstrap starting.
I20260924 16:05:38.766528 6223 tablet_bootstrap.cc:654] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:38.766592 6222 tablet_bootstrap.cc:654] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:38.767189 6221 tablet_bootstrap.cc:654] T 85bcefd82e5b46d5977700be540f2621 P 1e09099dc5684c09bcb34725ff82bc94: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:38.769356 6223 tablet_bootstrap.cc:492] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b: No bootstrap required, opened a new log
I20260924 16:05:38.769573 6223 ts_tablet_manager.cc:1397] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 16:05:38.769608 6222 tablet_bootstrap.cc:492] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026: No bootstrap required, opened a new log
I20260924 16:05:38.769814 6222 ts_tablet_manager.cc:1397] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 16:05:38.770313 6221 tablet_bootstrap.cc:492] T 85bcefd82e5b46d5977700be540f2621 P 1e09099dc5684c09bcb34725ff82bc94: No bootstrap required, opened a new log
I20260924 16:05:38.770469 6221 ts_tablet_manager.cc:1397] T 85bcefd82e5b46d5977700be540f2621 P 1e09099dc5684c09bcb34725ff82bc94: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 16:05:38.770566 6223 raft_consensus.cc:348] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:38.770848 6223 raft_consensus.cc:374] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:38.770713 6222 raft_consensus.cc:348] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:38.770934 6223 raft_consensus.cc:729] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b7eae966d1c4f11b8c6f273ea61c46b, State: Initialized, Role: FOLLOWER
I20260924 16:05:38.770989 6222 raft_consensus.cc:374] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:38.771152 6222 raft_consensus.cc:729] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57fe08e069c4e84a45f86322e3ed026, State: Initialized, Role: FOLLOWER
I20260924 16:05:38.771214 6223 consensus_queue.cc:260] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:38.771433 6222 consensus_queue.cc:260] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:38.771492 6221 raft_consensus.cc:348] T 85bcefd82e5b46d5977700be540f2621 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:38.771715 6221 raft_consensus.cc:374] T 85bcefd82e5b46d5977700be540f2621 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:38.771869 6223 ts_tablet_manager.cc:1428] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 16:05:38.772267 6222 ts_tablet_manager.cc:1428] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 16:05:38.772027 6228 tablet_replica.cc:439] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } } }
I20260924 16:05:38.771799 6221 raft_consensus.cc:729] T 85bcefd82e5b46d5977700be540f2621 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e09099dc5684c09bcb34725ff82bc94, State: Initialized, Role: FOLLOWER
I20260924 16:05:38.772555 6228 tablet_replica.cc:442] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:05:38.772753 6221 consensus_queue.cc:260] T 85bcefd82e5b46d5977700be540f2621 P 1e09099dc5684c09bcb34725ff82bc94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:38.773243 6229 tablet_replica.cc:439] T 85bcefd82e5b46d5977700be540f2621 P 1e09099dc5684c09bcb34725ff82bc94: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } } }
I20260924 16:05:38.773562 6229 tablet_replica.cc:442] T 85bcefd82e5b46d5977700be540f2621 P 1e09099dc5684c09bcb34725ff82bc94: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:05:38.773922 6227 tablet_replica.cc:439] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } } }
I20260924 16:05:38.774369 6227 tablet_replica.cc:442] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:05:38.774657 6221 ts_tablet_manager.cc:1428] T 85bcefd82e5b46d5977700be540f2621 P 1e09099dc5684c09bcb34725ff82bc94: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20260924 16:05:38.832926 6061 tablet.cc:2369] T 0303b8e6468b49adaf30881c626f6b28 P b57fe08e069c4e84a45f86322e3ed026: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:05:38.993223 6227 raft_consensus.cc:482] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:05:38.993479 6227 raft_consensus.cc:504] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:38.995220 6227 leader_election.cc:288] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e09099dc5684c09bcb34725ff82bc94 (127.3.214.193:36971), b57fe08e069c4e84a45f86322e3ed026 (127.3.214.194:38993)
I20260924 16:05:38.997350 6228 raft_consensus.cc:482] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:05:38.997644 6228 raft_consensus.cc:504] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:39.003914 6228 leader_election.cc:288] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e09099dc5684c09bcb34725ff82bc94 (127.3.214.193:36971), 2b7eae966d1c4f11b8c6f273ea61c46b (127.3.214.195:46149)
I20260924 16:05:39.008904 6220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:38.995935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:59175 (local address 127.3.214.193:36971)
0924 16:05:38.996362 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:38.996369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:38.996422 (+ 53us) server_negotiation.cc:407] Connection header received
0924 16:05:38.996590 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:38.996600 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:38.996729 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:38.996869 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:38.998698 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:38.999833 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.003730 (+ 3897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.005175 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.005338 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:39.005433 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:39.008430 (+ 2997us) server_negotiation.cc:299] Negotiation successful
0924 16:05:39.008597 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260924 16:05:39.009505 6216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:38.996818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:39623 (local address 127.3.214.194:38993)
0924 16:05:38.997025 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:38.997031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:38.997802 (+ 771us) server_negotiation.cc:407] Connection header received
0924 16:05:38.998242 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:38.998247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:38.998326 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:38.998385 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:38.999177 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.000384 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.006745 (+ 6361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.007975 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.008137 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:39.008218 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:39.008883 (+ 665us) server_negotiation.cc:299] Negotiation successful
0924 16:05:39.009167 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":83}
I20260924 16:05:39.010295 6243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:38.995517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:36971 (local address 127.3.214.195:59175)
0924 16:05:38.996147 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:38.996167 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:38.996303 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:38.997535 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:38.997542 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:38.997580 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:38.997852 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:38.997865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.000032 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.000038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:39.003498 (+ 3460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:39.003511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.005785 (+ 2274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.005793 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:39.006024 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:39.006066 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 16:05:39.009410 (+ 3344us) client_negotiation.cc:240] Negotiation successful
0924 16:05:39.009815 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":754,"thread_start_us":371,"threads_started":1}
I20260924 16:05:39.010689 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:38.996297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:38993 (local address 127.3.214.195:39623)
0924 16:05:38.997666 (+ 1369us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:38.997707 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:38.998025 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:38.998609 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:38.998614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:38.998650 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:38.998991 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:38.999005 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.001091 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.001097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:39.003270 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:39.003287 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.008412 (+ 5125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.008419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:39.008659 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:39.008699 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 16:05:39.010152 (+ 1453us) client_negotiation.cc:240] Negotiation successful
0924 16:05:39.010343 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1120,"thread_start_us":888,"threads_started":1}
I20260924 16:05:39.011644 6008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bcefd82e5b46d5977700be540f2621" candidate_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57fe08e069c4e84a45f86322e3ed026" is_pre_election: true
I20260924 16:05:39.011579 5864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bcefd82e5b46d5977700be540f2621" candidate_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e09099dc5684c09bcb34725ff82bc94" is_pre_election: true
I20260924 16:05:39.011981 6008 raft_consensus.cc:2454] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b7eae966d1c4f11b8c6f273ea61c46b in term 0.
I20260924 16:05:39.012141 5864 raft_consensus.cc:2454] T 85bcefd82e5b46d5977700be540f2621 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b7eae966d1c4f11b8c6f273ea61c46b in term 0.
I20260924 16:05:39.012651 6080 leader_election.cc:302] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b7eae966d1c4f11b8c6f273ea61c46b, b57fe08e069c4e84a45f86322e3ed026; no voters:
I20260924 16:05:39.013221 6227 raft_consensus.cc:2781] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:05:39.013370 6227 raft_consensus.cc:482] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:05:39.013481 6227 raft_consensus.cc:3037] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:39.015062 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.002682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:56793 (local address 127.3.214.195:46149)
0924 16:05:39.003565 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:39.003571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:39.007313 (+ 3742us) server_negotiation.cc:407] Connection header received
0924 16:05:39.007668 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:39.007674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:39.007756 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:39.007813 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:39.008709 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.009814 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.012618 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.013920 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.014079 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:39.014197 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:39.014594 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 16:05:39.014742 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752}
I20260924 16:05:39.015355 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.003184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:46149 (local address 127.3.214.194:56793)
0924 16:05:39.007161 (+ 3977us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:39.007187 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:39.007471 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:39.008029 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:39.008033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:39.008060 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:39.008525 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:39.008539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.010000 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.010005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:39.012434 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:39.012449 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.014171 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.014177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:39.014377 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:39.014413 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 16:05:39.014530 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 16:05:39.014704 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3779,"mutex_wait_us":3,"thread_start_us":3446,"threads_started":1}
I20260924 16:05:39.015763 6245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.001575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:36971 (local address 127.3.214.194:35539)
0924 16:05:39.003602 (+ 2027us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:39.003629 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:39.003818 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:39.008741 (+ 4923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:39.008748 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:39.008779 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:05:39.009285 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:39.009298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.011073 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.011079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:39.013161 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:39.013175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.014878 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.014885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:39.015065 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:39.015104 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 16:05:39.015193 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 16:05:39.015347 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1838,"mutex_wait_us":50,"thread_start_us":1964,"threads_started":1}
I20260924 16:05:39.016026 6246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.002997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:35539 (local address 127.3.214.193:36971)
0924 16:05:39.008410 (+ 5413us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:39.008415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:39.008439 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:05:39.008495 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:39.008499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:39.008575 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:39.009450 (+ 875us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:05:39.009715 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.010915 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.013392 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.014612 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.015416 (+ 804us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:39.015495 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:39.015548 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 16:05:39.015681 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5241,"thread_start_us":5009,"threads_started":1}
I20260924 16:05:39.016716 5864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bcefd82e5b46d5977700be540f2621" candidate_uuid: "b57fe08e069c4e84a45f86322e3ed026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e09099dc5684c09bcb34725ff82bc94" is_pre_election: true
I20260924 16:05:39.016723 6151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bcefd82e5b46d5977700be540f2621" candidate_uuid: "b57fe08e069c4e84a45f86322e3ed026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" is_pre_election: true
I20260924 16:05:39.016978 5864 raft_consensus.cc:2454] T 85bcefd82e5b46d5977700be540f2621 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b57fe08e069c4e84a45f86322e3ed026 in term 0.
I20260924 16:05:39.017470 6227 raft_consensus.cc:504] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:39.017546 5936 leader_election.cc:302] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e09099dc5684c09bcb34725ff82bc94, b57fe08e069c4e84a45f86322e3ed026; no voters:
I20260924 16:05:39.017975 6228 raft_consensus.cc:2781] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:05:39.018042 6151 raft_consensus.cc:2379] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b57fe08e069c4e84a45f86322e3ed026 in current term 1: Already voted for candidate 2b7eae966d1c4f11b8c6f273ea61c46b in this term.
I20260924 16:05:39.018154 6228 raft_consensus.cc:482] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:05:39.018270 6228 raft_consensus.cc:3037] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:39.018755 5864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bcefd82e5b46d5977700be540f2621" candidate_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e09099dc5684c09bcb34725ff82bc94"
I20260924 16:05:39.018997 5864 raft_consensus.cc:3037] T 85bcefd82e5b46d5977700be540f2621 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:39.019011 6008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bcefd82e5b46d5977700be540f2621" candidate_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57fe08e069c4e84a45f86322e3ed026"
I20260924 16:05:39.019321 6227 leader_election.cc:288] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b [CANDIDATE]: Term 1 election: Requested vote from peers 1e09099dc5684c09bcb34725ff82bc94 (127.3.214.193:36971), b57fe08e069c4e84a45f86322e3ed026 (127.3.214.194:38993)
I20260924 16:05:39.021260 6228 raft_consensus.cc:504] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:39.021723 6008 raft_consensus.cc:2379] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b7eae966d1c4f11b8c6f273ea61c46b in current term 1: Already voted for candidate b57fe08e069c4e84a45f86322e3ed026 in this term.
I20260924 16:05:39.021704 5864 raft_consensus.cc:2454] T 85bcefd82e5b46d5977700be540f2621 P 1e09099dc5684c09bcb34725ff82bc94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b7eae966d1c4f11b8c6f273ea61c46b in term 1.
I20260924 16:05:39.021993 6228 leader_election.cc:288] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026 [CANDIDATE]: Term 1 election: Requested vote from peers 1e09099dc5684c09bcb34725ff82bc94 (127.3.214.193:36971), 2b7eae966d1c4f11b8c6f273ea61c46b (127.3.214.195:46149)
I20260924 16:05:39.022182 6080 leader_election.cc:302] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e09099dc5684c09bcb34725ff82bc94, 2b7eae966d1c4f11b8c6f273ea61c46b; no voters:
I20260924 16:05:39.022679 5864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bcefd82e5b46d5977700be540f2621" candidate_uuid: "b57fe08e069c4e84a45f86322e3ed026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e09099dc5684c09bcb34725ff82bc94"
I20260924 16:05:39.022962 5864 raft_consensus.cc:2379] T 85bcefd82e5b46d5977700be540f2621 P 1e09099dc5684c09bcb34725ff82bc94 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b57fe08e069c4e84a45f86322e3ed026 in current term 1: Already voted for candidate 2b7eae966d1c4f11b8c6f273ea61c46b in this term.
I20260924 16:05:39.023053 6151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bcefd82e5b46d5977700be540f2621" candidate_uuid: "b57fe08e069c4e84a45f86322e3ed026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b"
I20260924 16:05:39.023269 6227 raft_consensus.cc:2781] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:39.026437 6227 raft_consensus.cc:686] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 1 LEADER]: Becoming Leader. State: Replica: 2b7eae966d1c4f11b8c6f273ea61c46b, State: Running, Role: LEADER
I20260924 16:05:39.027117 6227 consensus_queue.cc:237] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:39.030642 5939 leader_election.cc:302] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b57fe08e069c4e84a45f86322e3ed026; no voters: 1e09099dc5684c09bcb34725ff82bc94, 2b7eae966d1c4f11b8c6f273ea61c46b
I20260924 16:05:39.030503 6248 tablet_replica.cc:439] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b: TxnStatusTablet state changed. Reason: New leader 2b7eae966d1c4f11b8c6f273ea61c46b. Latest consensus state: current_term: 1 leader_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } } }
I20260924 16:05:39.030952 6248 tablet_replica.cc:442] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:05:39.031210 6228 raft_consensus.cc:2726] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 16:05:39.032007 6250 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 16:05:39.032333 6250 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 16:05:39.032815 5703 catalog_manager.cc:5495] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b7eae966d1c4f11b8c6f273ea61c46b (127.3.214.195). New cstate: current_term: 1 leader_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } health_report { overall_health: HEALTHY } } }
I20260924 16:05:39.083407 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.076123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:46149 (local address 127.0.0.1:51898)
0924 16:05:39.076464 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:39.076489 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:39.076702 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:39.077346 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:39.077350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:39.077369 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:05:39.077590 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:39.077598 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.079141 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.079146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:39.080252 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:39.080264 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.081360 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.081364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:39.081542 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:39.081629 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:05:39.082889 (+ 1260us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:05:39.082910 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 16:05:39.082976 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 16:05:39.083131 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 16:05:39.083500 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.076461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51898 (local address 127.3.214.195:46149)
0924 16:05:39.076738 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:39.076745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:39.076774 (+ 29us) server_negotiation.cc:407] Connection header received
0924 16:05:39.076933 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:39.076940 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:39.077094 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:39.077284 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:39.077751 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.078968 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.080405 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.081164 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.081302 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:39.081855 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:39.082769 (+ 914us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:39.082942 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:39.083002 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 16:05:39.083118 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260924 16:05:39.094053 5864 raft_consensus.cc:1261] T 85bcefd82e5b46d5977700be540f2621 P 1e09099dc5684c09bcb34725ff82bc94 [term 1 FOLLOWER]: Refusing update from remote peer 2b7eae966d1c4f11b8c6f273ea61c46b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:39.094317 6008 raft_consensus.cc:1261] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026 [term 1 FOLLOWER]: Refusing update from remote peer 2b7eae966d1c4f11b8c6f273ea61c46b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:39.095022 6248 consensus_queue.cc:1035] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:39.095525 6227 consensus_queue.cc:1035] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:39.104849 6228 tablet_replica.cc:439] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026: TxnStatusTablet state changed. Reason: New leader 2b7eae966d1c4f11b8c6f273ea61c46b. Latest consensus state: current_term: 1 leader_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } } }
I20260924 16:05:39.105295 6228 tablet_replica.cc:442] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:05:39.111294 6229 tablet_replica.cc:439] T 85bcefd82e5b46d5977700be540f2621 P 1e09099dc5684c09bcb34725ff82bc94: TxnStatusTablet state changed. Reason: New leader 2b7eae966d1c4f11b8c6f273ea61c46b. Latest consensus state: current_term: 1 leader_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } } }
I20260924 16:05:39.111686 6229 tablet_replica.cc:442] T 85bcefd82e5b46d5977700be540f2621 P 1e09099dc5684c09bcb34725ff82bc94: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:05:39.113399 6254 mvcc.cc:205] Tried to move back new op lower bound from 7332929286516822016 to 7332929286259974144. Current Snapshot: MvccSnapshot[applied={T|T < 7332929286516822016}]
I20260924 16:05:39.116204 6253 mvcc.cc:205] Tried to move back new op lower bound from 7332929286516822016 to 7332929286259974144. Current Snapshot: MvccSnapshot[applied={T|T < 7332929286516822016}]
I20260924 16:05:39.116709 6255 mvcc.cc:205] Tried to move back new op lower bound from 7332929286516822016 to 7332929286259974144. Current Snapshot: MvccSnapshot[applied={T|T < 7332929286516822016}]
I20260924 16:05:39.123629 6227 tablet_replica.cc:439] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } } }
I20260924 16:05:39.124074 6227 tablet_replica.cc:442] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:05:39.124383 6227 tablet_replica.cc:439] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } } }
I20260924 16:05:39.124661 6227 tablet_replica.cc:442] T 85bcefd82e5b46d5977700be540f2621 P 2b7eae966d1c4f11b8c6f273ea61c46b: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:05:39.125485 6229 tablet_replica.cc:439] T 85bcefd82e5b46d5977700be540f2621 P 1e09099dc5684c09bcb34725ff82bc94: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } } }
I20260924 16:05:39.125846 6229 tablet_replica.cc:442] T 85bcefd82e5b46d5977700be540f2621 P 1e09099dc5684c09bcb34725ff82bc94: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:05:39.127408 6228 tablet_replica.cc:439] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } } }
I20260924 16:05:39.127772 6228 tablet_replica.cc:442] T 85bcefd82e5b46d5977700be540f2621 P b57fe08e069c4e84a45f86322e3ed026: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:05:39.146647 6246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.139498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42040 (local address 127.3.214.193:36971)
0924 16:05:39.139765 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:39.139772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:39.140232 (+ 460us) server_negotiation.cc:407] Connection header received
0924 16:05:39.140505 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:39.140512 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:39.140645 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:39.140856 (+ 211us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:39.141313 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.142480 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.144008 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.144648 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.144800 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:39.145201 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:39.145992 (+ 791us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:39.146177 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:39.146241 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 16:05:39.146338 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 16:05:39.166620 6229 consensus_queue.cc:1035] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:39.167081 6266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.160230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:46149 (local address 127.3.214.193:42557)
0924 16:05:39.160900 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:39.160919 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:39.161077 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:39.161463 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:39.161467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:39.161476 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:05:39.161736 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:39.161744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.163016 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.163020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:39.164286 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:39.164299 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.165429 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.165435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:39.165653 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:39.165751 (+ 98us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:05:39.166367 (+ 616us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:05:39.166380 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 16:05:39.166458 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 16:05:39.166685 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":314,"threads_started":1}
I20260924 16:05:39.167454 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.160418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:42557 (local address 127.3.214.195:46149)
0924 16:05:39.160653 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:39.160660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:39.161019 (+ 359us) server_negotiation.cc:407] Connection header received
0924 16:05:39.161223 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:39.161229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:39.161326 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:39.161456 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:39.161950 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.162871 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.164457 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.165234 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.165399 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:39.166077 (+ 678us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:39.166249 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:39.166341 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:39.166560 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 16:05:39.166662 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 16:05:39.181223 6229 consensus_queue.cc:1035] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:39.228783 6246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.218787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:55071 (local address 127.3.214.193:36971)
0924 16:05:39.219034 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:39.219041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:39.220377 (+ 1336us) server_negotiation.cc:407] Connection header received
0924 16:05:39.220800 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:39.220806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:39.220959 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:39.221069 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:39.221851 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.223138 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.225670 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.226270 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.226427 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:39.227716 (+ 1289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:39.227923 (+ 207us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:39.228109 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:39.228263 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 16:05:39.228396 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 16:05:39.228974 6273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.218823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:36971 (local address 127.3.214.195:55071)
0924 16:05:39.220241 (+ 1418us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:39.220264 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:39.220423 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:39.221261 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:39.221266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:39.221278 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:05:39.221657 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:39.221666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.223384 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.223392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:39.225413 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:39.225441 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.227062 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.227068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:39.227283 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:39.227398 (+ 115us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:05:39.228111 (+ 713us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:05:39.228124 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 16:05:39.228225 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 16:05:39.228439 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"thread_start_us":921,"threads_started":1}
I20260924 16:05:39.280519 5703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:05:39.280898 5703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:05:39.286060 5703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884: Sending DeleteTablet for 3 replicas of tablet 0303b8e6468b49adaf30881c626f6b28
I20260924 16:05:39.287804 5986 tablet_service.cc:1514] Processing DeleteTablet for tablet 0303b8e6468b49adaf30881c626f6b28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:39 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:41540
I20260924 16:05:39.289623 6131 tablet_service.cc:1514] Processing DeleteTablet for tablet 0303b8e6468b49adaf30881c626f6b28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:39 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:51890
I20260924 16:05:39.298316 5844 tablet_service.cc:1514] Processing DeleteTablet for tablet 0303b8e6468b49adaf30881c626f6b28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:39 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:42028
I20260924 16:05:39.301000 5702 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:05:39.301975 5702 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:05:39.313236 5984 tablet_service.cc:1467] Processing CreateTablet for tablet 8a465bded1a442808b11c84d579edf09 (DEFAULT_TABLE table=test-table [id=3b39b547635a42b68074af6858526907]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:39.313864 5984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a465bded1a442808b11c84d579edf09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:39.313903 6130 tablet_service.cc:1467] Processing CreateTablet for tablet 8a465bded1a442808b11c84d579edf09 (DEFAULT_TABLE table=test-table [id=3b39b547635a42b68074af6858526907]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:39.314373 6130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a465bded1a442808b11c84d579edf09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:39.318698 5843 tablet_service.cc:1467] Processing CreateTablet for tablet 8a465bded1a442808b11c84d579edf09 (DEFAULT_TABLE table=test-table [id=3b39b547635a42b68074af6858526907]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:39.319243 5843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a465bded1a442808b11c84d579edf09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:39.323159 6278 tablet_replica.cc:330] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94: stopping tablet replica
I20260924 16:05:39.323711 6278 raft_consensus.cc:2229] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:05:39.324487 6278 raft_consensus.cc:2258] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:39.326586 6279 tablet_bootstrap.cc:492] T 8a465bded1a442808b11c84d579edf09 P 2b7eae966d1c4f11b8c6f273ea61c46b: Bootstrap starting.
I20260924 16:05:39.328855 6279 tablet_bootstrap.cc:654] T 8a465bded1a442808b11c84d579edf09 P 2b7eae966d1c4f11b8c6f273ea61c46b: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:39.333403 6278 ts_tablet_manager.cc:1905] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:39.339663 6276 tablet_replica.cc:330] T 0303b8e6468b49adaf30881c626f6b28 P b57fe08e069c4e84a45f86322e3ed026: stopping tablet replica
I20260924 16:05:39.340234 6276 raft_consensus.cc:2229] T 0303b8e6468b49adaf30881c626f6b28 P b57fe08e069c4e84a45f86322e3ed026 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:05:39.340647 6276 raft_consensus.cc:2258] T 0303b8e6468b49adaf30881c626f6b28 P b57fe08e069c4e84a45f86322e3ed026 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:39.341645 6279 tablet_bootstrap.cc:492] T 8a465bded1a442808b11c84d579edf09 P 2b7eae966d1c4f11b8c6f273ea61c46b: No bootstrap required, opened a new log
I20260924 16:05:39.341858 6279 ts_tablet_manager.cc:1397] T 8a465bded1a442808b11c84d579edf09 P 2b7eae966d1c4f11b8c6f273ea61c46b: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20260924 16:05:39.342957 6276 ts_tablet_manager.cc:1905] T 0303b8e6468b49adaf30881c626f6b28 P b57fe08e069c4e84a45f86322e3ed026: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:39.342864 6279 raft_consensus.cc:348] T 8a465bded1a442808b11c84d579edf09 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:39.343169 6279 raft_consensus.cc:374] T 8a465bded1a442808b11c84d579edf09 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:39.343277 6279 raft_consensus.cc:729] T 8a465bded1a442808b11c84d579edf09 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b7eae966d1c4f11b8c6f273ea61c46b, State: Initialized, Role: FOLLOWER
I20260924 16:05:39.343585 6279 consensus_queue.cc:260] T 8a465bded1a442808b11c84d579edf09 P 2b7eae966d1c4f11b8c6f273ea61c46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:39.345427 6279 ts_tablet_manager.cc:1428] T 8a465bded1a442808b11c84d579edf09 P 2b7eae966d1c4f11b8c6f273ea61c46b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 16:05:39.348489 6278 ts_tablet_manager.cc:1918] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:05:39.348719 6278 log.cc:1196] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-0/wal/wals/0303b8e6468b49adaf30881c626f6b28
I20260924 16:05:39.348989 6282 tablet_bootstrap.cc:492] T 8a465bded1a442808b11c84d579edf09 P 1e09099dc5684c09bcb34725ff82bc94: Bootstrap starting.
I20260924 16:05:39.349339 6278 ts_tablet_manager.cc:1939] T 0303b8e6468b49adaf30881c626f6b28 P 1e09099dc5684c09bcb34725ff82bc94: Deleting consensus metadata
I20260924 16:05:39.350172 6277 tablet_replica.cc:330] T 0303b8e6468b49adaf30881c626f6b28 P 2b7eae966d1c4f11b8c6f273ea61c46b: stopping tablet replica
I20260924 16:05:39.351131 6282 tablet_bootstrap.cc:654] T 8a465bded1a442808b11c84d579edf09 P 1e09099dc5684c09bcb34725ff82bc94: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:39.351481 5687 catalog_manager.cc:4841] TS 1e09099dc5684c09bcb34725ff82bc94 (127.3.214.193:36971): tablet 0303b8e6468b49adaf30881c626f6b28 (table test-table [id=941a43c0704d47d3814790d287699093]) successfully deleted
I20260924 16:05:39.352660 6277 raft_consensus.cc:2229] T 0303b8e6468b49adaf30881c626f6b28 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:05:39.353145 6280 tablet_bootstrap.cc:492] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026: Bootstrap starting.
I20260924 16:05:39.353345 6277 raft_consensus.cc:2258] T 0303b8e6468b49adaf30881c626f6b28 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:39.355841 6280 tablet_bootstrap.cc:654] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:39.356206 6277 ts_tablet_manager.cc:1905] T 0303b8e6468b49adaf30881c626f6b28 P 2b7eae966d1c4f11b8c6f273ea61c46b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:39.361471 6276 ts_tablet_manager.cc:1918] T 0303b8e6468b49adaf30881c626f6b28 P b57fe08e069c4e84a45f86322e3ed026: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:05:39.361708 6276 log.cc:1196] T 0303b8e6468b49adaf30881c626f6b28 P b57fe08e069c4e84a45f86322e3ed026: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/0303b8e6468b49adaf30881c626f6b28
I20260924 16:05:39.362321 6276 ts_tablet_manager.cc:1939] T 0303b8e6468b49adaf30881c626f6b28 P b57fe08e069c4e84a45f86322e3ed026: Deleting consensus metadata
I20260924 16:05:39.364687 5687 catalog_manager.cc:4841] TS b57fe08e069c4e84a45f86322e3ed026 (127.3.214.194:38993): tablet 0303b8e6468b49adaf30881c626f6b28 (table test-table [id=941a43c0704d47d3814790d287699093]) successfully deleted
I20260924 16:05:39.368033 6280 tablet_bootstrap.cc:492] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026: No bootstrap required, opened a new log
I20260924 16:05:39.368312 6280 ts_tablet_manager.cc:1397] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.004s
I20260924 16:05:39.369335 6280 raft_consensus.cc:348] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:39.369684 6280 raft_consensus.cc:374] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:39.369796 6280 raft_consensus.cc:729] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57fe08e069c4e84a45f86322e3ed026, State: Initialized, Role: FOLLOWER
I20260924 16:05:39.370127 6280 consensus_queue.cc:260] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:39.373236 6228 raft_consensus.cc:482] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:05:39.373454 6228 raft_consensus.cc:504] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:39.374181 6228 leader_election.cc:288] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e09099dc5684c09bcb34725ff82bc94 (127.3.214.193:36971), 2b7eae966d1c4f11b8c6f273ea61c46b (127.3.214.195:46149)
I20260924 16:05:39.375015 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a465bded1a442808b11c84d579edf09" candidate_uuid: "b57fe08e069c4e84a45f86322e3ed026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e09099dc5684c09bcb34725ff82bc94" is_pre_election: true
I20260924 16:05:39.375573 6280 ts_tablet_manager.cc:1428] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
W20260924 16:05:39.376044 5936 leader_election.cc:341] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e09099dc5684c09bcb34725ff82bc94 (127.3.214.193:36971): Illegal state: must be running to vote when last-logged opid is not known
I20260924 16:05:39.376930 6282 tablet_bootstrap.cc:492] T 8a465bded1a442808b11c84d579edf09 P 1e09099dc5684c09bcb34725ff82bc94: No bootstrap required, opened a new log
I20260924 16:05:39.377156 6282 ts_tablet_manager.cc:1397] T 8a465bded1a442808b11c84d579edf09 P 1e09099dc5684c09bcb34725ff82bc94: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.010s
I20260924 16:05:39.377071 6151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a465bded1a442808b11c84d579edf09" candidate_uuid: "b57fe08e069c4e84a45f86322e3ed026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" is_pre_election: true
I20260924 16:05:39.377368 6151 raft_consensus.cc:2454] T 8a465bded1a442808b11c84d579edf09 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b57fe08e069c4e84a45f86322e3ed026 in term 0.
I20260924 16:05:39.377779 6277 ts_tablet_manager.cc:1918] T 0303b8e6468b49adaf30881c626f6b28 P 2b7eae966d1c4f11b8c6f273ea61c46b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:05:39.377851 5939 leader_election.cc:302] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b7eae966d1c4f11b8c6f273ea61c46b, b57fe08e069c4e84a45f86322e3ed026; no voters: 1e09099dc5684c09bcb34725ff82bc94
I20260924 16:05:39.378252 6282 raft_consensus.cc:348] T 8a465bded1a442808b11c84d579edf09 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:39.378547 6282 raft_consensus.cc:374] T 8a465bded1a442808b11c84d579edf09 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:39.378649 6282 raft_consensus.cc:729] T 8a465bded1a442808b11c84d579edf09 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e09099dc5684c09bcb34725ff82bc94, State: Initialized, Role: FOLLOWER
I20260924 16:05:39.378818 6277 log.cc:1196] T 0303b8e6468b49adaf30881c626f6b28 P 2b7eae966d1c4f11b8c6f273ea61c46b: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/0303b8e6468b49adaf30881c626f6b28
I20260924 16:05:39.378978 6282 consensus_queue.cc:260] T 8a465bded1a442808b11c84d579edf09 P 1e09099dc5684c09bcb34725ff82bc94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:39.379937 6277 ts_tablet_manager.cc:1939] T 0303b8e6468b49adaf30881c626f6b28 P 2b7eae966d1c4f11b8c6f273ea61c46b: Deleting consensus metadata
I20260924 16:05:39.380566 6282 ts_tablet_manager.cc:1428] T 8a465bded1a442808b11c84d579edf09 P 1e09099dc5684c09bcb34725ff82bc94: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 16:05:39.381253 6228 raft_consensus.cc:2781] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:05:39.381383 6228 raft_consensus.cc:482] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:05:39.381501 6228 raft_consensus.cc:3037] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:39.384893 6228 raft_consensus.cc:504] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:39.385785 6228 leader_election.cc:288] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026 [CANDIDATE]: Term 1 election: Requested vote from peers 1e09099dc5684c09bcb34725ff82bc94 (127.3.214.193:36971), 2b7eae966d1c4f11b8c6f273ea61c46b (127.3.214.195:46149)
I20260924 16:05:39.386485 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a465bded1a442808b11c84d579edf09" candidate_uuid: "b57fe08e069c4e84a45f86322e3ed026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e09099dc5684c09bcb34725ff82bc94"
I20260924 16:05:39.386790 5863 raft_consensus.cc:3037] T 8a465bded1a442808b11c84d579edf09 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:39.389169 5690 catalog_manager.cc:4841] TS 2b7eae966d1c4f11b8c6f273ea61c46b (127.3.214.195:46149): tablet 0303b8e6468b49adaf30881c626f6b28 (table test-table [id=941a43c0704d47d3814790d287699093]) successfully deleted
I20260924 16:05:39.390154 6151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a465bded1a442808b11c84d579edf09" candidate_uuid: "b57fe08e069c4e84a45f86322e3ed026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b"
I20260924 16:05:39.390275 5863 raft_consensus.cc:2454] T 8a465bded1a442808b11c84d579edf09 P 1e09099dc5684c09bcb34725ff82bc94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b57fe08e069c4e84a45f86322e3ed026 in term 1.
I20260924 16:05:39.390468 6151 raft_consensus.cc:3037] T 8a465bded1a442808b11c84d579edf09 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:39.390844 5936 leader_election.cc:302] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e09099dc5684c09bcb34725ff82bc94, b57fe08e069c4e84a45f86322e3ed026; no voters:
I20260924 16:05:39.391201 6228 raft_consensus.cc:2781] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:39.394029 6151 raft_consensus.cc:2454] T 8a465bded1a442808b11c84d579edf09 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b57fe08e069c4e84a45f86322e3ed026 in term 1.
I20260924 16:05:39.396380 6228 raft_consensus.cc:686] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026 [term 1 LEADER]: Becoming Leader. State: Replica: b57fe08e069c4e84a45f86322e3ed026, State: Running, Role: LEADER
I20260924 16:05:39.397313 6228 consensus_queue.cc:237] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:39.401989 5702 catalog_manager.cc:5495] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026 reported cstate change: term changed from 0 to 1, leader changed from <none> to b57fe08e069c4e84a45f86322e3ed026 (127.3.214.194). New cstate: current_term: 1 leader_uuid: "b57fe08e069c4e84a45f86322e3ed026" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } health_report { overall_health: UNKNOWN } } }
I20260924 16:05:39.440795 6216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.432867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41544 (local address 127.3.214.194:38993)
0924 16:05:39.433088 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:39.433095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:39.433119 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:05:39.433250 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:39.433256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:39.433392 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:39.433588 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:39.434069 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.435214 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.437440 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.438179 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.438328 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:39.438936 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:39.439761 (+ 825us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:39.439926 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:39.440105 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 16:05:39.440211 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 16:05:39.484056 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.469669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:47263 (local address 127.3.214.195:46149)
0924 16:05:39.469910 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:39.469917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:39.476436 (+ 6519us) server_negotiation.cc:407] Connection header received
0924 16:05:39.477006 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:39.477013 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:39.477111 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:39.477177 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:39.477996 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.479276 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.481466 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.482203 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.482314 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:39.483093 (+ 779us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:39.483288 (+ 195us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:39.483352 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:39.483609 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 16:05:39.483715 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 16:05:39.484990 6289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.470144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:46149 (local address 127.3.214.194:47263)
0924 16:05:39.476269 (+ 6125us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:39.476298 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:39.476799 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:39.477401 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:39.477406 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:39.477420 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:05:39.477781 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:39.477794 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.479515 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.479521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:39.481258 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:39.481277 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.482541 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.482548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:39.482751 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:39.482840 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:05:39.483447 (+ 607us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:05:39.483467 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 16:05:39.484386 (+ 919us) client_negotiation.cc:240] Negotiation successful
0924 16:05:39.484568 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5863,"thread_start_us":6321,"threads_started":1}
I20260924 16:05:39.499311 6151 raft_consensus.cc:1261] T 8a465bded1a442808b11c84d579edf09 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 1 FOLLOWER]: Refusing update from remote peer b57fe08e069c4e84a45f86322e3ed026: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:39.500368 6285 consensus_queue.cc:1035] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:39.502362 5863 raft_consensus.cc:1261] T 8a465bded1a442808b11c84d579edf09 P 1e09099dc5684c09bcb34725ff82bc94 [term 1 FOLLOWER]: Refusing update from remote peer b57fe08e069c4e84a45f86322e3ed026: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:39.503502 6285 consensus_queue.cc:1035] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:39.571694 6216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.557736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:52895 (local address 127.3.214.194:38993)
0924 16:05:39.558117 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:39.558124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:39.558145 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:05:39.558219 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:39.558224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:39.558335 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:39.558426 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:39.559149 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.560298 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.564780 (+ 4482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.565621 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.565767 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:39.566564 (+ 797us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:39.566747 (+ 183us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:39.566826 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:39.571193 (+ 4367us) server_negotiation.cc:299] Negotiation successful
0924 16:05:39.571368 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260924 16:05:39.572192 6273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:39.557278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:38993 (local address 127.3.214.195:52895)
0924 16:05:39.557553 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:39.557579 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:39.557726 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:39.558607 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:39.558613 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:39.558627 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:05:39.558879 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:39.558892 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.560847 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.560853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:39.562620 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:39.562643 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:39.565989 (+ 3346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:39.565997 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:39.566259 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:39.566322 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:05:39.570386 (+ 4064us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:05:39.570408 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:05:39.570499 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 16:05:39.570686 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
W20260924 16:05:39.573725 6254 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 16:05:41.215456 5703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:05:41.215785 5703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:05:41.221325 5703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884: Sending DeleteTablet for 3 replicas of tablet 8a465bded1a442808b11c84d579edf09
I20260924 16:05:41.222637 5843 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a465bded1a442808b11c84d579edf09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:41 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:42028
I20260924 16:05:41.222766 5984 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a465bded1a442808b11c84d579edf09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:41 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:41540
I20260924 16:05:41.223203 6131 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a465bded1a442808b11c84d579edf09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:41 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:51890
I20260924 16:05:41.223747 6317 tablet_replica.cc:330] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026: stopping tablet replica
I20260924 16:05:41.223964 6318 tablet_replica.cc:330] T 8a465bded1a442808b11c84d579edf09 P 2b7eae966d1c4f11b8c6f273ea61c46b: stopping tablet replica
I20260924 16:05:41.224016 6317 raft_consensus.cc:2229] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:05:41.224282 6318 raft_consensus.cc:2229] T 8a465bded1a442808b11c84d579edf09 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:05:41.224611 6318 raft_consensus.cc:2258] T 8a465bded1a442808b11c84d579edf09 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:41.224779 6317 raft_consensus.cc:2258] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:41.225816 6316 tablet_replica.cc:330] T 8a465bded1a442808b11c84d579edf09 P 1e09099dc5684c09bcb34725ff82bc94: stopping tablet replica
I20260924 16:05:41.225958 6318 ts_tablet_manager.cc:1905] T 8a465bded1a442808b11c84d579edf09 P 2b7eae966d1c4f11b8c6f273ea61c46b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:41.226063 6316 raft_consensus.cc:2229] T 8a465bded1a442808b11c84d579edf09 P 1e09099dc5684c09bcb34725ff82bc94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:05:41.226419 6316 raft_consensus.cc:2258] T 8a465bded1a442808b11c84d579edf09 P 1e09099dc5684c09bcb34725ff82bc94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:41.226480 6317 ts_tablet_manager.cc:1905] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:41.227480 6316 ts_tablet_manager.cc:1905] T 8a465bded1a442808b11c84d579edf09 P 1e09099dc5684c09bcb34725ff82bc94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:41.233563 6318 ts_tablet_manager.cc:1918] T 8a465bded1a442808b11c84d579edf09 P 2b7eae966d1c4f11b8c6f273ea61c46b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 16:05:41.233592 6317 ts_tablet_manager.cc:1918] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 16:05:41.233726 6317 log.cc:1196] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/8a465bded1a442808b11c84d579edf09
I20260924 16:05:41.233875 6318 log.cc:1196] T 8a465bded1a442808b11c84d579edf09 P 2b7eae966d1c4f11b8c6f273ea61c46b: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/8a465bded1a442808b11c84d579edf09
I20260924 16:05:41.234215 6317 ts_tablet_manager.cc:1939] T 8a465bded1a442808b11c84d579edf09 P b57fe08e069c4e84a45f86322e3ed026: Deleting consensus metadata
I20260924 16:05:41.234467 6318 ts_tablet_manager.cc:1939] T 8a465bded1a442808b11c84d579edf09 P 2b7eae966d1c4f11b8c6f273ea61c46b: Deleting consensus metadata
I20260924 16:05:41.234483 6316 ts_tablet_manager.cc:1918] T 8a465bded1a442808b11c84d579edf09 P 1e09099dc5684c09bcb34725ff82bc94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 16:05:41.234659 6316 log.cc:1196] T 8a465bded1a442808b11c84d579edf09 P 1e09099dc5684c09bcb34725ff82bc94: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-0/wal/wals/8a465bded1a442808b11c84d579edf09
I20260924 16:05:41.235082 6316 ts_tablet_manager.cc:1939] T 8a465bded1a442808b11c84d579edf09 P 1e09099dc5684c09bcb34725ff82bc94: Deleting consensus metadata
I20260924 16:05:41.235584 5687 catalog_manager.cc:4841] TS b57fe08e069c4e84a45f86322e3ed026 (127.3.214.194:38993): tablet 8a465bded1a442808b11c84d579edf09 (table test-table [id=3b39b547635a42b68074af6858526907]) successfully deleted
I20260924 16:05:41.235868 5690 catalog_manager.cc:4841] TS 2b7eae966d1c4f11b8c6f273ea61c46b (127.3.214.195:46149): tablet 8a465bded1a442808b11c84d579edf09 (table test-table [id=3b39b547635a42b68074af6858526907]) successfully deleted
I20260924 16:05:41.236910 5687 catalog_manager.cc:4841] TS 1e09099dc5684c09bcb34725ff82bc94 (127.3.214.193:36971): tablet 8a465bded1a442808b11c84d579edf09 (table test-table [id=3b39b547635a42b68074af6858526907]) successfully deleted
Sep 24 16:05:41 dist-test-slave-1gc3 krb5kdc[5658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265941, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 16:05:41 dist-test-slave-1gc3 krb5kdc[5658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265941, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:41.288259 6326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:41.269854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43864 (local address 127.3.214.254:43765)
0924 16:05:41.271190 (+ 1336us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:41.271197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:41.274297 (+ 3100us) server_negotiation.cc:407] Connection header received
0924 16:05:41.274523 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:41.274529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:41.274607 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:41.274705 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:41.276168 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:41.276992 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:41.278275 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:41.278714 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:41.278836 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:41.282908 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:41.282913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:41.282917 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:41.282958 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:41.285574 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:41.286287 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:41.286291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:41.286294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:41.286442 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:41.286807 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:41.286811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:41.286813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:41.287105 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:41.287339 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:41.287667 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 16:05:41.287894 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1073,"thread_start_us":847,"threads_started":1}
I20260924 16:05:41.293040 5703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:05:41.293805 5703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:05:41.301942 5984 tablet_service.cc:1467] Processing CreateTablet for tablet 93b6b080a6f84e898a67d63cd5bf80e6 (DEFAULT_TABLE table=test-table [id=368058bc9511473a96a143ac9e725c02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:41.301942 5843 tablet_service.cc:1467] Processing CreateTablet for tablet 93b6b080a6f84e898a67d63cd5bf80e6 (DEFAULT_TABLE table=test-table [id=368058bc9511473a96a143ac9e725c02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:41.302353 6131 tablet_service.cc:1467] Processing CreateTablet for tablet 93b6b080a6f84e898a67d63cd5bf80e6 (DEFAULT_TABLE table=test-table [id=368058bc9511473a96a143ac9e725c02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:41.302495 5984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93b6b080a6f84e898a67d63cd5bf80e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:41.302505 5843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93b6b080a6f84e898a67d63cd5bf80e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:41.302829 6131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93b6b080a6f84e898a67d63cd5bf80e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:41.309401 6327 tablet_bootstrap.cc:492] T 93b6b080a6f84e898a67d63cd5bf80e6 P 2b7eae966d1c4f11b8c6f273ea61c46b: Bootstrap starting.
I20260924 16:05:41.310264 6328 tablet_bootstrap.cc:492] T 93b6b080a6f84e898a67d63cd5bf80e6 P b57fe08e069c4e84a45f86322e3ed026: Bootstrap starting.
I20260924 16:05:41.311165 6329 tablet_bootstrap.cc:492] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94: Bootstrap starting.
I20260924 16:05:41.312212 6328 tablet_bootstrap.cc:654] T 93b6b080a6f84e898a67d63cd5bf80e6 P b57fe08e069c4e84a45f86322e3ed026: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:41.312222 6327 tablet_bootstrap.cc:654] T 93b6b080a6f84e898a67d63cd5bf80e6 P 2b7eae966d1c4f11b8c6f273ea61c46b: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:41.312949 6329 tablet_bootstrap.cc:654] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:41.315001 6328 tablet_bootstrap.cc:492] T 93b6b080a6f84e898a67d63cd5bf80e6 P b57fe08e069c4e84a45f86322e3ed026: No bootstrap required, opened a new log
I20260924 16:05:41.315138 6327 tablet_bootstrap.cc:492] T 93b6b080a6f84e898a67d63cd5bf80e6 P 2b7eae966d1c4f11b8c6f273ea61c46b: No bootstrap required, opened a new log
I20260924 16:05:41.315201 6328 ts_tablet_manager.cc:1397] T 93b6b080a6f84e898a67d63cd5bf80e6 P b57fe08e069c4e84a45f86322e3ed026: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260924 16:05:41.315281 6327 ts_tablet_manager.cc:1397] T 93b6b080a6f84e898a67d63cd5bf80e6 P 2b7eae966d1c4f11b8c6f273ea61c46b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:05:41.315320 6329 tablet_bootstrap.cc:492] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94: No bootstrap required, opened a new log
I20260924 16:05:41.315465 6329 ts_tablet_manager.cc:1397] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 16:05:41.316088 6327 raft_consensus.cc:348] T 93b6b080a6f84e898a67d63cd5bf80e6 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:41.316088 6328 raft_consensus.cc:348] T 93b6b080a6f84e898a67d63cd5bf80e6 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:41.316342 6328 raft_consensus.cc:374] T 93b6b080a6f84e898a67d63cd5bf80e6 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:41.316342 6327 raft_consensus.cc:374] T 93b6b080a6f84e898a67d63cd5bf80e6 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:41.316430 6327 raft_consensus.cc:729] T 93b6b080a6f84e898a67d63cd5bf80e6 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b7eae966d1c4f11b8c6f273ea61c46b, State: Initialized, Role: FOLLOWER
I20260924 16:05:41.316430 6328 raft_consensus.cc:729] T 93b6b080a6f84e898a67d63cd5bf80e6 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57fe08e069c4e84a45f86322e3ed026, State: Initialized, Role: FOLLOWER
I20260924 16:05:41.316669 6328 consensus_queue.cc:260] T 93b6b080a6f84e898a67d63cd5bf80e6 P b57fe08e069c4e84a45f86322e3ed026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:41.316669 6327 consensus_queue.cc:260] T 93b6b080a6f84e898a67d63cd5bf80e6 P 2b7eae966d1c4f11b8c6f273ea61c46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:41.316613 6329 raft_consensus.cc:348] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:41.316958 6329 raft_consensus.cc:374] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:41.317059 6329 raft_consensus.cc:729] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e09099dc5684c09bcb34725ff82bc94, State: Initialized, Role: FOLLOWER
I20260924 16:05:41.317271 6328 ts_tablet_manager.cc:1428] T 93b6b080a6f84e898a67d63cd5bf80e6 P b57fe08e069c4e84a45f86322e3ed026: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 16:05:41.317260 6327 ts_tablet_manager.cc:1428] T 93b6b080a6f84e898a67d63cd5bf80e6 P 2b7eae966d1c4f11b8c6f273ea61c46b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 16:05:41.317364 6329 consensus_queue.cc:260] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:41.319200 6329 ts_tablet_manager.cc:1428] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 16:05:41.583545 6333 raft_consensus.cc:482] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:05:41.583892 6333 raft_consensus.cc:504] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:41.584745 6333 leader_election.cc:288] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b57fe08e069c4e84a45f86322e3ed026 (127.3.214.194:38993), 2b7eae966d1c4f11b8c6f273ea61c46b (127.3.214.195:46149)
I20260924 16:05:41.585678 6151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b6b080a6f84e898a67d63cd5bf80e6" candidate_uuid: "1e09099dc5684c09bcb34725ff82bc94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" is_pre_election: true
I20260924 16:05:41.585678 6007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b6b080a6f84e898a67d63cd5bf80e6" candidate_uuid: "1e09099dc5684c09bcb34725ff82bc94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57fe08e069c4e84a45f86322e3ed026" is_pre_election: true
I20260924 16:05:41.586010 6007 raft_consensus.cc:2454] T 93b6b080a6f84e898a67d63cd5bf80e6 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e09099dc5684c09bcb34725ff82bc94 in term 0.
I20260924 16:05:41.586011 6151 raft_consensus.cc:2454] T 93b6b080a6f84e898a67d63cd5bf80e6 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e09099dc5684c09bcb34725ff82bc94 in term 0.
I20260924 16:05:41.586685 5796 leader_election.cc:302] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1e09099dc5684c09bcb34725ff82bc94, 2b7eae966d1c4f11b8c6f273ea61c46b, b57fe08e069c4e84a45f86322e3ed026; no voters:
I20260924 16:05:41.587077 6333 raft_consensus.cc:2781] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:05:41.587225 6333 raft_consensus.cc:482] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:05:41.587325 6333 raft_consensus.cc:3037] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:41.589915 6333 raft_consensus.cc:504] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:41.590690 6333 leader_election.cc:288] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94 [CANDIDATE]: Term 1 election: Requested vote from peers b57fe08e069c4e84a45f86322e3ed026 (127.3.214.194:38993), 2b7eae966d1c4f11b8c6f273ea61c46b (127.3.214.195:46149)
I20260924 16:05:41.591548 6151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b6b080a6f84e898a67d63cd5bf80e6" candidate_uuid: "1e09099dc5684c09bcb34725ff82bc94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b"
I20260924 16:05:41.591548 6007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b6b080a6f84e898a67d63cd5bf80e6" candidate_uuid: "1e09099dc5684c09bcb34725ff82bc94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57fe08e069c4e84a45f86322e3ed026"
I20260924 16:05:41.591838 6151 raft_consensus.cc:3037] T 93b6b080a6f84e898a67d63cd5bf80e6 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:41.591851 6007 raft_consensus.cc:3037] T 93b6b080a6f84e898a67d63cd5bf80e6 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:41.595233 6007 raft_consensus.cc:2454] T 93b6b080a6f84e898a67d63cd5bf80e6 P b57fe08e069c4e84a45f86322e3ed026 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e09099dc5684c09bcb34725ff82bc94 in term 1.
I20260924 16:05:41.595238 6151 raft_consensus.cc:2454] T 93b6b080a6f84e898a67d63cd5bf80e6 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e09099dc5684c09bcb34725ff82bc94 in term 1.
I20260924 16:05:41.595970 5793 leader_election.cc:302] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1e09099dc5684c09bcb34725ff82bc94, 2b7eae966d1c4f11b8c6f273ea61c46b, b57fe08e069c4e84a45f86322e3ed026; no voters:
I20260924 16:05:41.596401 6333 raft_consensus.cc:2781] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:41.596948 6333 raft_consensus.cc:686] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94 [term 1 LEADER]: Becoming Leader. State: Replica: 1e09099dc5684c09bcb34725ff82bc94, State: Running, Role: LEADER
I20260924 16:05:41.597293 6333 consensus_queue.cc:237] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:41.600534 5702 catalog_manager.cc:5495] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e09099dc5684c09bcb34725ff82bc94 (127.3.214.193). New cstate: current_term: 1 leader_uuid: "1e09099dc5684c09bcb34725ff82bc94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } health_report { overall_health: UNKNOWN } } }
I20260924 16:05:41.627516 6337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:41.621563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42044 (local address 127.3.214.193:36971)
0924 16:05:41.622374 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:41.622379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:41.622403 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:05:41.622472 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:41.622476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:41.622553 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:41.622657 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:41.623044 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:41.623907 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:41.625369 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:41.625841 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:41.625946 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:41.626567 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:41.626730 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:41.626826 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:41.627000 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 16:05:41.627082 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":474,"threads_started":1}
I20260924 16:05:41.638404 6007 raft_consensus.cc:1261] T 93b6b080a6f84e898a67d63cd5bf80e6 P b57fe08e069c4e84a45f86322e3ed026 [term 1 FOLLOWER]: Refusing update from remote peer 1e09099dc5684c09bcb34725ff82bc94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:41.638432 6151 raft_consensus.cc:1261] T 93b6b080a6f84e898a67d63cd5bf80e6 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 1 FOLLOWER]: Refusing update from remote peer 1e09099dc5684c09bcb34725ff82bc94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:41.639019 6334 consensus_queue.cc:1035] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:41.639403 6333 consensus_queue.cc:1035] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:41.701963 5702 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:05:41.702323 5702 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:05:41.706135 5702 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884: Sending DeleteTablet for 3 replicas of tablet 93b6b080a6f84e898a67d63cd5bf80e6
I20260924 16:05:41.707293 5843 tablet_service.cc:1514] Processing DeleteTablet for tablet 93b6b080a6f84e898a67d63cd5bf80e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:41 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:42028
I20260924 16:05:41.707372 5984 tablet_service.cc:1514] Processing DeleteTablet for tablet 93b6b080a6f84e898a67d63cd5bf80e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:41 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:41540
I20260924 16:05:41.707414 6131 tablet_service.cc:1514] Processing DeleteTablet for tablet 93b6b080a6f84e898a67d63cd5bf80e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:41 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:51890
I20260924 16:05:41.707742 6318 tablet_replica.cc:330] T 93b6b080a6f84e898a67d63cd5bf80e6 P 2b7eae966d1c4f11b8c6f273ea61c46b: stopping tablet replica
I20260924 16:05:41.707758 6317 tablet_replica.cc:330] T 93b6b080a6f84e898a67d63cd5bf80e6 P b57fe08e069c4e84a45f86322e3ed026: stopping tablet replica
I20260924 16:05:41.707762 6316 tablet_replica.cc:330] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94: stopping tablet replica
I20260924 16:05:41.707988 6317 raft_consensus.cc:2229] T 93b6b080a6f84e898a67d63cd5bf80e6 P b57fe08e069c4e84a45f86322e3ed026 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:05:41.707988 6316 raft_consensus.cc:2229] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:05:41.707983 6318 raft_consensus.cc:2229] T 93b6b080a6f84e898a67d63cd5bf80e6 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:05:41.708225 6318 raft_consensus.cc:2258] T 93b6b080a6f84e898a67d63cd5bf80e6 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:41.708357 6317 raft_consensus.cc:2258] T 93b6b080a6f84e898a67d63cd5bf80e6 P b57fe08e069c4e84a45f86322e3ed026 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:41.708595 6316 raft_consensus.cc:2258] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:41.708973 5702 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 16:05:41.709708 6318 ts_tablet_manager.cc:1905] T 93b6b080a6f84e898a67d63cd5bf80e6 P 2b7eae966d1c4f11b8c6f273ea61c46b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 16:05:41.709749 5702 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:05:41.709952 6317 ts_tablet_manager.cc:1905] T 93b6b080a6f84e898a67d63cd5bf80e6 P b57fe08e069c4e84a45f86322e3ed026: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:41.709998 6316 ts_tablet_manager.cc:1905] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:41.718297 6317 ts_tablet_manager.cc:1918] T 93b6b080a6f84e898a67d63cd5bf80e6 P b57fe08e069c4e84a45f86322e3ed026: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:05:41.718585 6317 log.cc:1196] T 93b6b080a6f84e898a67d63cd5bf80e6 P b57fe08e069c4e84a45f86322e3ed026: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/93b6b080a6f84e898a67d63cd5bf80e6
I20260924 16:05:41.718808 6318 ts_tablet_manager.cc:1918] T 93b6b080a6f84e898a67d63cd5bf80e6 P 2b7eae966d1c4f11b8c6f273ea61c46b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:05:41.719009 6318 log.cc:1196] T 93b6b080a6f84e898a67d63cd5bf80e6 P 2b7eae966d1c4f11b8c6f273ea61c46b: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/93b6b080a6f84e898a67d63cd5bf80e6
I20260924 16:05:41.719148 6317 ts_tablet_manager.cc:1939] T 93b6b080a6f84e898a67d63cd5bf80e6 P b57fe08e069c4e84a45f86322e3ed026: Deleting consensus metadata
I20260924 16:05:41.719457 6318 ts_tablet_manager.cc:1939] T 93b6b080a6f84e898a67d63cd5bf80e6 P 2b7eae966d1c4f11b8c6f273ea61c46b: Deleting consensus metadata
I20260924 16:05:41.719504 6316 ts_tablet_manager.cc:1918] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:05:41.719792 6316 log.cc:1196] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-0/wal/wals/93b6b080a6f84e898a67d63cd5bf80e6
I20260924 16:05:41.720393 6316 ts_tablet_manager.cc:1939] T 93b6b080a6f84e898a67d63cd5bf80e6 P 1e09099dc5684c09bcb34725ff82bc94: Deleting consensus metadata
I20260924 16:05:41.721098 6131 tablet_service.cc:1467] Processing CreateTablet for tablet 6257eec2ff874ab48693d96a8b4f5c30 (DEFAULT_TABLE table=test-table [id=282dde7572294a979edf5bc38eb8d123]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:41.721299 5984 tablet_service.cc:1467] Processing CreateTablet for tablet 6257eec2ff874ab48693d96a8b4f5c30 (DEFAULT_TABLE table=test-table [id=282dde7572294a979edf5bc38eb8d123]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:41.721753 6131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6257eec2ff874ab48693d96a8b4f5c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:41.721771 5984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6257eec2ff874ab48693d96a8b4f5c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:41.722218 5687 catalog_manager.cc:4841] TS 1e09099dc5684c09bcb34725ff82bc94 (127.3.214.193:36971): tablet 93b6b080a6f84e898a67d63cd5bf80e6 (table test-table [id=368058bc9511473a96a143ac9e725c02]) successfully deleted
I20260924 16:05:41.722318 5843 tablet_service.cc:1467] Processing CreateTablet for tablet 6257eec2ff874ab48693d96a8b4f5c30 (DEFAULT_TABLE table=test-table [id=282dde7572294a979edf5bc38eb8d123]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:05:41.722884 5843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6257eec2ff874ab48693d96a8b4f5c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:41.723368 5687 catalog_manager.cc:4841] TS b57fe08e069c4e84a45f86322e3ed026 (127.3.214.194:38993): tablet 93b6b080a6f84e898a67d63cd5bf80e6 (table test-table [id=368058bc9511473a96a143ac9e725c02]) successfully deleted
I20260924 16:05:41.725512 5690 catalog_manager.cc:4841] TS 2b7eae966d1c4f11b8c6f273ea61c46b (127.3.214.195:46149): tablet 93b6b080a6f84e898a67d63cd5bf80e6 (table test-table [id=368058bc9511473a96a143ac9e725c02]) successfully deleted
I20260924 16:05:41.729055 6329 tablet_bootstrap.cc:492] T 6257eec2ff874ab48693d96a8b4f5c30 P 1e09099dc5684c09bcb34725ff82bc94: Bootstrap starting.
I20260924 16:05:41.729274 6327 tablet_bootstrap.cc:492] T 6257eec2ff874ab48693d96a8b4f5c30 P 2b7eae966d1c4f11b8c6f273ea61c46b: Bootstrap starting.
I20260924 16:05:41.729754 6328 tablet_bootstrap.cc:492] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026: Bootstrap starting.
I20260924 16:05:41.731354 6329 tablet_bootstrap.cc:654] T 6257eec2ff874ab48693d96a8b4f5c30 P 1e09099dc5684c09bcb34725ff82bc94: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:41.731354 6327 tablet_bootstrap.cc:654] T 6257eec2ff874ab48693d96a8b4f5c30 P 2b7eae966d1c4f11b8c6f273ea61c46b: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:41.731449 6328 tablet_bootstrap.cc:654] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:41.734263 6329 tablet_bootstrap.cc:492] T 6257eec2ff874ab48693d96a8b4f5c30 P 1e09099dc5684c09bcb34725ff82bc94: No bootstrap required, opened a new log
I20260924 16:05:41.734423 6328 tablet_bootstrap.cc:492] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026: No bootstrap required, opened a new log
I20260924 16:05:41.734458 6329 ts_tablet_manager.cc:1397] T 6257eec2ff874ab48693d96a8b4f5c30 P 1e09099dc5684c09bcb34725ff82bc94: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:05:41.734620 6328 ts_tablet_manager.cc:1397] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 16:05:41.735471 6327 tablet_bootstrap.cc:492] T 6257eec2ff874ab48693d96a8b4f5c30 P 2b7eae966d1c4f11b8c6f273ea61c46b: No bootstrap required, opened a new log
I20260924 16:05:41.735425 6328 raft_consensus.cc:348] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:41.735671 6327 ts_tablet_manager.cc:1397] T 6257eec2ff874ab48693d96a8b4f5c30 P 2b7eae966d1c4f11b8c6f273ea61c46b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 16:05:41.735466 6329 raft_consensus.cc:348] T 6257eec2ff874ab48693d96a8b4f5c30 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:41.735734 6328 raft_consensus.cc:374] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:41.735801 6329 raft_consensus.cc:374] T 6257eec2ff874ab48693d96a8b4f5c30 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:41.735826 6328 raft_consensus.cc:729] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57fe08e069c4e84a45f86322e3ed026, State: Initialized, Role: FOLLOWER
I20260924 16:05:41.735906 6329 raft_consensus.cc:729] T 6257eec2ff874ab48693d96a8b4f5c30 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e09099dc5684c09bcb34725ff82bc94, State: Initialized, Role: FOLLOWER
I20260924 16:05:41.736088 6328 consensus_queue.cc:260] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:41.736217 6329 consensus_queue.cc:260] T 6257eec2ff874ab48693d96a8b4f5c30 P 1e09099dc5684c09bcb34725ff82bc94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:41.736559 6327 raft_consensus.cc:348] T 6257eec2ff874ab48693d96a8b4f5c30 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:41.736680 6328 ts_tablet_manager.cc:1428] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 16:05:41.736843 6327 raft_consensus.cc:374] T 6257eec2ff874ab48693d96a8b4f5c30 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:41.736930 6327 raft_consensus.cc:729] T 6257eec2ff874ab48693d96a8b4f5c30 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b7eae966d1c4f11b8c6f273ea61c46b, State: Initialized, Role: FOLLOWER
I20260924 16:05:41.737313 6327 consensus_queue.cc:260] T 6257eec2ff874ab48693d96a8b4f5c30 P 2b7eae966d1c4f11b8c6f273ea61c46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:41.738224 6329 ts_tablet_manager.cc:1428] T 6257eec2ff874ab48693d96a8b4f5c30 P 1e09099dc5684c09bcb34725ff82bc94: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 16:05:41.739390 6327 ts_tablet_manager.cc:1428] T 6257eec2ff874ab48693d96a8b4f5c30 P 2b7eae966d1c4f11b8c6f273ea61c46b: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260924 16:05:41.950073 6294 raft_consensus.cc:482] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:05:41.950333 6294 raft_consensus.cc:504] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:41.951112 6294 leader_election.cc:288] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e09099dc5684c09bcb34725ff82bc94 (127.3.214.193:36971), 2b7eae966d1c4f11b8c6f273ea61c46b (127.3.214.195:46149)
I20260924 16:05:41.951879 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6257eec2ff874ab48693d96a8b4f5c30" candidate_uuid: "b57fe08e069c4e84a45f86322e3ed026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e09099dc5684c09bcb34725ff82bc94" is_pre_election: true
I20260924 16:05:41.952157 5863 raft_consensus.cc:2454] T 6257eec2ff874ab48693d96a8b4f5c30 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b57fe08e069c4e84a45f86322e3ed026 in term 0.
I20260924 16:05:41.952035 6151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6257eec2ff874ab48693d96a8b4f5c30" candidate_uuid: "b57fe08e069c4e84a45f86322e3ed026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" is_pre_election: true
I20260924 16:05:41.952375 6151 raft_consensus.cc:2454] T 6257eec2ff874ab48693d96a8b4f5c30 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b57fe08e069c4e84a45f86322e3ed026 in term 0.
I20260924 16:05:41.952696 5936 leader_election.cc:302] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e09099dc5684c09bcb34725ff82bc94, b57fe08e069c4e84a45f86322e3ed026; no voters:
I20260924 16:05:41.953055 6294 raft_consensus.cc:2781] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:05:41.953195 6294 raft_consensus.cc:482] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:05:41.953311 6294 raft_consensus.cc:3037] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:41.955783 6294 raft_consensus.cc:504] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:41.956491 6294 leader_election.cc:288] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026 [CANDIDATE]: Term 1 election: Requested vote from peers 1e09099dc5684c09bcb34725ff82bc94 (127.3.214.193:36971), 2b7eae966d1c4f11b8c6f273ea61c46b (127.3.214.195:46149)
I20260924 16:05:41.957170 6151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6257eec2ff874ab48693d96a8b4f5c30" candidate_uuid: "b57fe08e069c4e84a45f86322e3ed026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b"
I20260924 16:05:41.957238 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6257eec2ff874ab48693d96a8b4f5c30" candidate_uuid: "b57fe08e069c4e84a45f86322e3ed026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e09099dc5684c09bcb34725ff82bc94"
I20260924 16:05:41.957394 6151 raft_consensus.cc:3037] T 6257eec2ff874ab48693d96a8b4f5c30 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:41.957538 5863 raft_consensus.cc:3037] T 6257eec2ff874ab48693d96a8b4f5c30 P 1e09099dc5684c09bcb34725ff82bc94 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:41.959887 5863 raft_consensus.cc:2454] T 6257eec2ff874ab48693d96a8b4f5c30 P 1e09099dc5684c09bcb34725ff82bc94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b57fe08e069c4e84a45f86322e3ed026 in term 1.
I20260924 16:05:41.959909 6151 raft_consensus.cc:2454] T 6257eec2ff874ab48693d96a8b4f5c30 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b57fe08e069c4e84a45f86322e3ed026 in term 1.
I20260924 16:05:41.960546 5936 leader_election.cc:302] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e09099dc5684c09bcb34725ff82bc94, b57fe08e069c4e84a45f86322e3ed026; no voters:
I20260924 16:05:41.960920 6294 raft_consensus.cc:2781] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:41.961354 6294 raft_consensus.cc:686] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026 [term 1 LEADER]: Becoming Leader. State: Replica: b57fe08e069c4e84a45f86322e3ed026, State: Running, Role: LEADER
I20260924 16:05:41.961681 6294 consensus_queue.cc:237] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } }
I20260924 16:05:41.964977 5701 catalog_manager.cc:5495] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026 reported cstate change: term changed from 0 to 1, leader changed from <none> to b57fe08e069c4e84a45f86322e3ed026 (127.3.214.194). New cstate: current_term: 1 leader_uuid: "b57fe08e069c4e84a45f86322e3ed026" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b57fe08e069c4e84a45f86322e3ed026" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 38993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 } health_report { overall_health: UNKNOWN } } }
I20260924 16:05:42.042169 6348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:42.036653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41556 (local address 127.3.214.194:38993)
0924 16:05:42.037209 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:42.037214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:42.037237 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:05:42.037314 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:42.037317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:42.037388 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:42.037481 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:05:42.037928 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:42.038705 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:42.040198 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:42.040708 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:42.040823 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:42.041354 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:05:42.041512 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:05:42.041609 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:42.041725 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 16:05:42.041800 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":288,"threads_started":1}
I20260924 16:05:42.054045 5863 raft_consensus.cc:1261] T 6257eec2ff874ab48693d96a8b4f5c30 P 1e09099dc5684c09bcb34725ff82bc94 [term 1 FOLLOWER]: Refusing update from remote peer b57fe08e069c4e84a45f86322e3ed026: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:42.054051 6151 raft_consensus.cc:1261] T 6257eec2ff874ab48693d96a8b4f5c30 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 1 FOLLOWER]: Refusing update from remote peer b57fe08e069c4e84a45f86322e3ed026: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:05:42.054833 6294 consensus_queue.cc:1035] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b7eae966d1c4f11b8c6f273ea61c46b" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:05:42.055189 6346 consensus_queue.cc:1035] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e09099dc5684c09bcb34725ff82bc94" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 36971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 16:05:42.080268 6298 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 16:05:43.679035 5703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:05:43.679406 5703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:05:43.685097 5703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97c89026d44e402093c6635a028db884: Sending DeleteTablet for 3 replicas of tablet 6257eec2ff874ab48693d96a8b4f5c30
I20260924 16:05:43.686393 5843 tablet_service.cc:1514] Processing DeleteTablet for tablet 6257eec2ff874ab48693d96a8b4f5c30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:43 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:42028
I20260924 16:05:43.686482 5984 tablet_service.cc:1514] Processing DeleteTablet for tablet 6257eec2ff874ab48693d96a8b4f5c30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:43 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:41540
I20260924 16:05:43.686599 6131 tablet_service.cc:1514] Processing DeleteTablet for tablet 6257eec2ff874ab48693d96a8b4f5c30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:05:43 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:51890
I20260924 16:05:43.687052 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 5776
I20260924 16:05:43.698801 6379 tablet_replica.cc:330] T 6257eec2ff874ab48693d96a8b4f5c30 P 1e09099dc5684c09bcb34725ff82bc94: stopping tablet replica
I20260924 16:05:43.699126 6381 tablet_replica.cc:330] T 6257eec2ff874ab48693d96a8b4f5c30 P 2b7eae966d1c4f11b8c6f273ea61c46b: stopping tablet replica
I20260924 16:05:43.699349 6379 raft_consensus.cc:2229] T 6257eec2ff874ab48693d96a8b4f5c30 P 1e09099dc5684c09bcb34725ff82bc94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:05:43.699470 6381 raft_consensus.cc:2229] T 6257eec2ff874ab48693d96a8b4f5c30 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:05:43.699762 6379 raft_consensus.cc:2258] T 6257eec2ff874ab48693d96a8b4f5c30 P 1e09099dc5684c09bcb34725ff82bc94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:43.699790 6381 raft_consensus.cc:2258] T 6257eec2ff874ab48693d96a8b4f5c30 P 2b7eae966d1c4f11b8c6f273ea61c46b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:43.700513 6380 tablet_replica.cc:330] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026: stopping tablet replica
I20260924 16:05:43.701506 6381 ts_tablet_manager.cc:1905] T 6257eec2ff874ab48693d96a8b4f5c30 P 2b7eae966d1c4f11b8c6f273ea61c46b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:43.701630 6379 ts_tablet_manager.cc:1905] T 6257eec2ff874ab48693d96a8b4f5c30 P 1e09099dc5684c09bcb34725ff82bc94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:43.702381 6380 raft_consensus.cc:2229] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:05:43.702950 6380 raft_consensus.cc:2258] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:05:43.704335 6380 ts_tablet_manager.cc:1905] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:05:43.708838 6379 ts_tablet_manager.cc:1918] T 6257eec2ff874ab48693d96a8b4f5c30 P 1e09099dc5684c09bcb34725ff82bc94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 16:05:43.709033 6379 log.cc:1196] T 6257eec2ff874ab48693d96a8b4f5c30 P 1e09099dc5684c09bcb34725ff82bc94: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-0/wal/wals/6257eec2ff874ab48693d96a8b4f5c30
I20260924 16:05:43.709086 6381 ts_tablet_manager.cc:1918] T 6257eec2ff874ab48693d96a8b4f5c30 P 2b7eae966d1c4f11b8c6f273ea61c46b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 16:05:43.709288 6381 log.cc:1196] T 6257eec2ff874ab48693d96a8b4f5c30 P 2b7eae966d1c4f11b8c6f273ea61c46b: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/6257eec2ff874ab48693d96a8b4f5c30
I20260924 16:05:43.709520 6379 ts_tablet_manager.cc:1939] T 6257eec2ff874ab48693d96a8b4f5c30 P 1e09099dc5684c09bcb34725ff82bc94: Deleting consensus metadata
I20260924 16:05:43.709805 6381 ts_tablet_manager.cc:1939] T 6257eec2ff874ab48693d96a8b4f5c30 P 2b7eae966d1c4f11b8c6f273ea61c46b: Deleting consensus metadata
I20260924 16:05:43.710937 5687 catalog_manager.cc:4841] TS 1e09099dc5684c09bcb34725ff82bc94 (127.3.214.193:36971): tablet 6257eec2ff874ab48693d96a8b4f5c30 (table test-table [id=282dde7572294a979edf5bc38eb8d123]) successfully deleted
I20260924 16:05:43.711340 5690 catalog_manager.cc:4841] TS 2b7eae966d1c4f11b8c6f273ea61c46b (127.3.214.195:46149): tablet 6257eec2ff874ab48693d96a8b4f5c30 (table test-table [id=282dde7572294a979edf5bc38eb8d123]) successfully deleted
I20260924 16:05:43.711794 6380 ts_tablet_manager.cc:1918] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 16:05:43.711923 6380 log.cc:1196] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/6257eec2ff874ab48693d96a8b4f5c30
I20260924 16:05:43.712396 6380 ts_tablet_manager.cc:1939] T 6257eec2ff874ab48693d96a8b4f5c30 P b57fe08e069c4e84a45f86322e3ed026: Deleting consensus metadata
Sep 24 16:05:43 dist-test-slave-1gc3 krb5kdc[5658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265941, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.214.193@KRBTEST.COM
I20260924 16:05:43.713730 5687 catalog_manager.cc:4841] TS b57fe08e069c4e84a45f86322e3ed026 (127.3.214.194:38993): tablet 6257eec2ff874ab48693d96a8b4f5c30 (table test-table [id=282dde7572294a979edf5bc38eb8d123]) successfully deleted
I20260924 16:05:43.722168 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:43.688822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42050 (local address 127.3.214.193:36971)
0924 16:05:43.696631 (+ 7809us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:43.696639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:43.696684 (+ 45us) server_negotiation.cc:407] Connection header received
0924 16:05:43.696783 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:43.696791 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:43.696905 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:43.697065 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:43.699327 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:43.700511 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:43.702569 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:43.703246 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:43.703380 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:43.714087 (+ 10707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:43.714151 (+ 64us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:43.714183 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:43.714254 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:43.718042 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:43.719145 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:43.719159 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:43.719162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:43.719390 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:43.719983 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:43.719988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:43.719990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:43.720481 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:43.720859 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:43.721424 (+ 565us) server_negotiation.cc:299] Negotiation successful
0924 16:05:43.721687 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7596,"thread_start_us":7330,"threads_started":1}
W20260924 16:05:43.723083 5910 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42050
W20260924 16:05:43.723901 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 5776: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:05:43.724231 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 5776
I20260924 16:05:43.748353 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 5919
Sep 24 16:05:43 dist-test-slave-1gc3 krb5kdc[5658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265941, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.214.194@KRBTEST.COM
I20260924 16:05:43.767675 6385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:43.749277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41562 (local address 127.3.214.194:38993)
0924 16:05:43.750061 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:43.750066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:43.750091 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:05:43.750204 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:43.750209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:43.750294 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:43.750411 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:43.751755 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:43.752592 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:43.753997 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:43.754560 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:43.754709 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:43.759669 (+ 4960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:43.759696 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:43.759734 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:43.759790 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:43.763689 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:43.764614 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:43.764624 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:43.764627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:43.764883 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:43.765362 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:43.765366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:43.765368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:43.766146 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:43.766537 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:43.766933 (+ 396us) server_negotiation.cc:299] Negotiation successful
0924 16:05:43.767252 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":443,"threads_started":1}
W20260924 16:05:43.768666 6054 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41562
W20260924 16:05:43.769600 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 5919: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:05:43.769779 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 5919
I20260924 16:05:43.793493 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 6064
Sep 24 16:05:43 dist-test-slave-1gc3 krb5kdc[5658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265941, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.214.195@KRBTEST.COM
I20260924 16:05:43.812809 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:43.794399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51906 (local address 127.3.214.195:46149)
0924 16:05:43.794962 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:43.794967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:43.794988 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:05:43.795050 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:43.795054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:43.795119 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:43.795229 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:43.796748 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:43.797563 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:43.798999 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:43.799493 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:43.799620 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:43.805422 (+ 5802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:43.805456 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:43.805496 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:43.805579 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:43.809324 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:43.810328 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:43.810337 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:43.810340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:43.810519 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:43.811081 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:43.811084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:43.811086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:43.811547 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:43.811811 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:43.812146 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 16:05:43.812387 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":299,"threads_started":1}
W20260924 16:05:43.813705 6194 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51906
W20260924 16:05:43.814563 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 6064: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:05:43.814781 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 6064
I20260924 16:05:43.839931 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 5671
I20260924 16:05:43.841321 5744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:43.980613 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 5671
2026-09-24T16:05:44Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9261 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:05:44 dist-test-slave-1gc3 krb5kdc[6390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:05:44 dist-test-slave-1gc3 krb5kdc[6390](info): set up 2 sockets
Sep 24 16:05:44 dist-test-slave-1gc3 krb5kdc[6390](info): commencing operation
krb5kdc: starting...
W20260924 16:05:45.016294 3931 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:05:45 dist-test-slave-1gc3 krb5kdc[6390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:05:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:05:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:05:45.245609 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:39915
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:39915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:45.646746 6405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:45.647156 6405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:45.647284 6405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:45.655460 6405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:05:45.655615 6405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:45.655669 6405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:45.655817 6405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:05:45.655880 6405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:05:45.667595 6405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:39915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:39915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:45.669569 6405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:45.671499 6405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:45.686976 6410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:45.687170 6413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:45.687150 6411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:45.688663 6405 server_base.cc:1034] running on GCE node
I20260924 16:05:45.689659 6405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:45.691514 6405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:45.692808 6405 hybrid_clock.cc:648] HybridClock initialized: now 1790265945692721 us; error 74 us; skew 500 ppm
I20260924 16:05:45.693310 6405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:45 dist-test-slave-1gc3 krb5kdc[6390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265945, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:45.697145 6405 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:05:45.699120 6405 webserver.cc:460] Webserver started at http://127.3.214.254:34503/ using document root <none> and password file <none>
I20260924 16:05:45.699967 6405 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:45.700099 6405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:45.700438 6405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:45.703822 6405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "8ad690fd656941dfa3783fedc09f7bc0"
format_stamp: "Formatted at 2026-09-24 16:05:45 on dist-test-slave-1gc3"
server_key: "77dd9bdc7c200f570bd28ab37ed517fa"
server_key_iv: "b694fdbf6f85a7753e9eb6c46c112f4b"
server_key_version: "encryptionkey@0"
I20260924 16:05:45.704677 6405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "8ad690fd656941dfa3783fedc09f7bc0"
format_stamp: "Formatted at 2026-09-24 16:05:45 on dist-test-slave-1gc3"
server_key: "77dd9bdc7c200f570bd28ab37ed517fa"
server_key_iv: "b694fdbf6f85a7753e9eb6c46c112f4b"
server_key_version: "encryptionkey@0"
I20260924 16:05:45.711472 6405 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 16:05:45.716147 6420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:45.718022 6405 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:05:45.718298 6405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "8ad690fd656941dfa3783fedc09f7bc0"
format_stamp: "Formatted at 2026-09-24 16:05:45 on dist-test-slave-1gc3"
server_key: "77dd9bdc7c200f570bd28ab37ed517fa"
server_key_iv: "b694fdbf6f85a7753e9eb6c46c112f4b"
server_key_version: "encryptionkey@0"
I20260924 16:05:45.718532 6405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:45.739878 6405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:45.744716 6405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:45.744992 6405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:45.765369 6405 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:39915
I20260924 16:05:45.765427 6474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:39915 every 8 connection(s)
I20260924 16:05:45.766991 6405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:05:45.770041 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 6405
I20260924 16:05:45.770295 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:05:45.770974 3931 external_mini_cluster.cc:1428] Setting key 5df7b1f6560a257d21f8a09954ff3dd0
I20260924 16:05:45.772619 6475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 16:05:45 dist-test-slave-1gc3 krb5kdc[6390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:45.791407 6475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ad690fd656941dfa3783fedc09f7bc0: Bootstrap starting.
I20260924 16:05:45.797267 6475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ad690fd656941dfa3783fedc09f7bc0: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:45.799134 6475 log.cc:824] T 00000000000000000000000000000000 P 8ad690fd656941dfa3783fedc09f7bc0: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:45.801990 6478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:45.773613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40068 (local address 127.3.214.254:39915)
0924 16:05:45.774688 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:45.774708 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:45.774791 (+ 83us) server_negotiation.cc:407] Connection header received
0924 16:05:45.777050 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:45.777086 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:45.777792 (+ 706us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:45.778406 (+ 614us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:45.779990 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:45.782486 (+ 2496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:45.784058 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:45.785366 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:45.785679 (+ 313us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:45.790707 (+ 5028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:45.790725 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:45.790757 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:45.790835 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:45.795592 (+ 4757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:45.796722 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:45.796734 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:45.796745 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:45.797029 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:45.797519 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:45.797525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:45.797528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:45.798534 (+ 1006us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:45.798934 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:45.799514 (+ 580us) server_negotiation.cc:299] Negotiation successful
0924 16:05:45.800216 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":445,"threads_started":1}
I20260924 16:05:45.804001 6475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ad690fd656941dfa3783fedc09f7bc0: No bootstrap required, opened a new log
I20260924 16:05:45.814445 6475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ad690fd656941dfa3783fedc09f7bc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad690fd656941dfa3783fedc09f7bc0" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39915 } }
I20260924 16:05:45.815021 6475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ad690fd656941dfa3783fedc09f7bc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:45.815141 6475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ad690fd656941dfa3783fedc09f7bc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad690fd656941dfa3783fedc09f7bc0, State: Initialized, Role: FOLLOWER
I20260924 16:05:45.815984 6475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ad690fd656941dfa3783fedc09f7bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad690fd656941dfa3783fedc09f7bc0" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39915 } }
I20260924 16:05:45.816272 6475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ad690fd656941dfa3783fedc09f7bc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:05:45.816398 6475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ad690fd656941dfa3783fedc09f7bc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:05:45.816566 6475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ad690fd656941dfa3783fedc09f7bc0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:45.819519 6475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ad690fd656941dfa3783fedc09f7bc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad690fd656941dfa3783fedc09f7bc0" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39915 } }
I20260924 16:05:45.820317 6475 leader_election.cc:302] T 00000000000000000000000000000000 P 8ad690fd656941dfa3783fedc09f7bc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ad690fd656941dfa3783fedc09f7bc0; no voters:
I20260924 16:05:45.821063 6475 leader_election.cc:288] T 00000000000000000000000000000000 P 8ad690fd656941dfa3783fedc09f7bc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:05:45.821303 6480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ad690fd656941dfa3783fedc09f7bc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:45.822197 6480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ad690fd656941dfa3783fedc09f7bc0 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad690fd656941dfa3783fedc09f7bc0, State: Running, Role: LEADER
I20260924 16:05:45.822930 6480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ad690fd656941dfa3783fedc09f7bc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad690fd656941dfa3783fedc09f7bc0" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39915 } }
I20260924 16:05:45.823797 6475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ad690fd656941dfa3783fedc09f7bc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:05:45.827708 6481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ad690fd656941dfa3783fedc09f7bc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ad690fd656941dfa3783fedc09f7bc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad690fd656941dfa3783fedc09f7bc0" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39915 } } }
I20260924 16:05:45.828493 6481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ad690fd656941dfa3783fedc09f7bc0 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:45.827509 6482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ad690fd656941dfa3783fedc09f7bc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ad690fd656941dfa3783fedc09f7bc0. Latest consensus state: current_term: 1 leader_uuid: "8ad690fd656941dfa3783fedc09f7bc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad690fd656941dfa3783fedc09f7bc0" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39915 } } }
I20260924 16:05:45.830001 6482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ad690fd656941dfa3783fedc09f7bc0 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:45.832769 6487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:05:45.839847 6487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:05:45.850452 6487 catalog_manager.cc:1347] Generated new cluster ID: 15496796f31348548f85bfe1ccf1e1dd
I20260924 16:05:45.850617 6487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:05:45.863819 6487 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:05:45.866379 6487 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:05:45.881969 6487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ad690fd656941dfa3783fedc09f7bc0: Generated new TSK 0
I20260924 16:05:45.883232 6487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.214.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
I20260924 16:05:45.946648 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39915
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:46.357420 6502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:46.357841 6502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:46.357972 6502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:46.366235 6502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:46.366425 6502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:46.366607 6502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:05:46.378299 6502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:46.380111 6502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:46.381755 6502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:46.398021 6510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:46.398173 6507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:46.398121 6508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:46.398937 6502 server_base.cc:1034] running on GCE node
I20260924 16:05:46.399784 6502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:46.401068 6502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:46.402352 6502 hybrid_clock.cc:648] HybridClock initialized: now 1790265946402284 us; error 55 us; skew 500 ppm
I20260924 16:05:46.402840 6502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:46 dist-test-slave-1gc3 krb5kdc[6390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265946, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:46.406775 6502 init.cc:377] Logged in from keytab as kudu/127.3.214.193@KRBTEST.COM (short username kudu)
I20260924 16:05:46.408867 6502 webserver.cc:460] Webserver started at http://127.3.214.193:39353/ using document root <none> and password file <none>
I20260924 16:05:46.409755 6502 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:46.409880 6502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:46.410305 6502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:46.413931 6502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "1e4e340cdb414744812c5026e70d2f87"
format_stamp: "Formatted at 2026-09-24 16:05:46 on dist-test-slave-1gc3"
server_key: "307efcffc203abbad1b69759607f6765"
server_key_iv: "ccf2ce5987be8d49f303714d583e1498"
server_key_version: "encryptionkey@0"
I20260924 16:05:46.414924 6502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "1e4e340cdb414744812c5026e70d2f87"
format_stamp: "Formatted at 2026-09-24 16:05:46 on dist-test-slave-1gc3"
server_key: "307efcffc203abbad1b69759607f6765"
server_key_iv: "ccf2ce5987be8d49f303714d583e1498"
server_key_version: "encryptionkey@0"
I20260924 16:05:46.421259 6502 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 16:05:46.425760 6517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:46.427698 6502 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:05:46.427932 6502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "1e4e340cdb414744812c5026e70d2f87"
format_stamp: "Formatted at 2026-09-24 16:05:46 on dist-test-slave-1gc3"
server_key: "307efcffc203abbad1b69759607f6765"
server_key_iv: "ccf2ce5987be8d49f303714d583e1498"
server_key_version: "encryptionkey@0"
I20260924 16:05:46.428180 6502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:46.446398 6502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:46.451877 6502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:46.452207 6502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:46.453459 6502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:46.455197 6502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:46.455286 6502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:46.455427 6502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:46.455488 6502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:46.493221 6502 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:46829
I20260924 16:05:46.493317 6634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:46829 every 8 connection(s)
I20260924 16:05:46.494925 6502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:05:46.505184 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 6502
I20260924 16:05:46.505398 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:05:46.506055 3931 external_mini_cluster.cc:1428] Setting key 1a54d6d5e8298190fb9cbd734a554d4f
Sep 24 16:05:46 dist-test-slave-1gc3 krb5kdc[6390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265946, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:46.521996 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:46.498075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:36831 (local address 127.3.214.254:39915)
0924 16:05:46.498610 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:46.498615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:46.500171 (+ 1556us) server_negotiation.cc:407] Connection header received
0924 16:05:46.502594 (+ 2423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:46.502600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:46.502696 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:46.502800 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:46.505038 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:46.505997 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:46.508424 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:46.509055 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:46.509211 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:46.514188 (+ 4977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:46.514195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:46.514200 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:46.514253 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:46.517733 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:46.518896 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:46.518902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:46.518905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:46.519079 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:46.519586 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:46.519590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:46.519593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:46.520002 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:46.520170 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:46.521258 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0924 16:05:46.521570 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":258,"threads_started":1}
I20260924 16:05:46.523018 6638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:46.498725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39915 (local address 127.3.214.193:36831)
0924 16:05:46.499896 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:46.500020 (+ 124us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:46.502230 (+ 2210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:46.503113 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:46.503122 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:46.503905 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:46.504714 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:46.504747 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:46.506266 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:46.506272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:46.508111 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:46.508128 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:46.509359 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:46.509366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:46.509701 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:46.510602 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:46.510634 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:46.513761 (+ 3127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:46.518004 (+ 4243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:46.518016 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:46.518036 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:46.518719 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:46.519196 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:46.519199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:46.519201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:46.519462 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:46.520180 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:46.520188 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:46.520633 (+ 445us) client_negotiation.cc:769] Sending connection context
0924 16:05:46.521051 (+ 418us) client_negotiation.cc:240] Negotiation successful
0924 16:05:46.521588 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":331,"threads_started":1}
I20260924 16:05:46.524971 6635 heartbeater.cc:347] Connected to a master server at 127.3.214.254:39915
I20260924 16:05:46.525336 6635 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:46.526262 6635 heartbeater.cc:511] Master 127.3.214.254:39915 requested a full tablet report, sending...
I20260924 16:05:46.529552 6438 ts_manager.cc:194] Registered new tserver with Master: 1e4e340cdb414744812c5026e70d2f87 (127.3.214.193:46829)
WARNING: no policy specified for kudu/127.3.214.194@KRBTEST.COM; defaulting to no policy
I20260924 16:05:46.531755 6438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:36831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
I20260924 16:05:46.561306 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39915
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:46.964319 6641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:46.964725 6641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:46.964877 6641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:46.972991 6641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:46.973179 6641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:46.973351 6641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:05:46.985013 6641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:46.986905 6641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:46.988703 6641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:47.005105 6648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:47.005019 6647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:47.005690 6650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:47.006690 6641 server_base.cc:1034] running on GCE node
I20260924 16:05:47.007481 6641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:47.008807 6641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:47.010159 6641 hybrid_clock.cc:648] HybridClock initialized: now 1790265947010030 us; error 83 us; skew 500 ppm
I20260924 16:05:47.010600 6641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:47 dist-test-slave-1gc3 krb5kdc[6390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265947, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:47.014325 6641 init.cc:377] Logged in from keytab as kudu/127.3.214.194@KRBTEST.COM (short username kudu)
I20260924 16:05:47.016259 6641 webserver.cc:460] Webserver started at http://127.3.214.194:40519/ using document root <none> and password file <none>
I20260924 16:05:47.017135 6641 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:47.017277 6641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:47.017633 6641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:47.021494 6641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "546f475a10d046dd8fb89f551689644e"
format_stamp: "Formatted at 2026-09-24 16:05:47 on dist-test-slave-1gc3"
server_key: "ec7a75a1c367bfa2702078a5c0fe3c60"
server_key_iv: "6ec624d53bcbe701944059ba027d4392"
server_key_version: "encryptionkey@0"
I20260924 16:05:47.022616 6641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "546f475a10d046dd8fb89f551689644e"
format_stamp: "Formatted at 2026-09-24 16:05:47 on dist-test-slave-1gc3"
server_key: "ec7a75a1c367bfa2702078a5c0fe3c60"
server_key_iv: "6ec624d53bcbe701944059ba027d4392"
server_key_version: "encryptionkey@0"
I20260924 16:05:47.029695 6641 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260924 16:05:47.034348 6657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:47.036111 6641 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260924 16:05:47.036312 6641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "546f475a10d046dd8fb89f551689644e"
format_stamp: "Formatted at 2026-09-24 16:05:47 on dist-test-slave-1gc3"
server_key: "ec7a75a1c367bfa2702078a5c0fe3c60"
server_key_iv: "6ec624d53bcbe701944059ba027d4392"
server_key_version: "encryptionkey@0"
I20260924 16:05:47.036538 6641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:47.108256 6641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:47.112946 6641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:47.113257 6641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:47.114506 6641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:47.116143 6641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:47.116227 6641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:47.116359 6641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:47.116413 6641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:47.153611 6641 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:46643
I20260924 16:05:47.153749 6771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:46643 every 8 connection(s)
I20260924 16:05:47.155063 6641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:05:47.157817 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 6641
I20260924 16:05:47.158030 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:05:47.158716 3931 external_mini_cluster.cc:1428] Setting key c6505f8be94d95885a0a528fead4164a
Sep 24 16:05:47 dist-test-slave-1gc3 krb5kdc[6390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265947, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:47.182124 6774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:47.158406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:50815 (local address 127.3.214.254:39915)
0924 16:05:47.159006 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:47.159013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:47.160138 (+ 1125us) server_negotiation.cc:407] Connection header received
0924 16:05:47.162881 (+ 2743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:47.162886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:47.162968 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:47.163093 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:47.165746 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:47.166633 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:47.169034 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:47.169541 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:47.169683 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:47.174634 (+ 4951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:47.174642 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:47.174647 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:47.174699 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:47.177608 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:47.179003 (+ 1395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:47.179010 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:47.179014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:47.179195 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:47.179758 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:47.179763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:47.179767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:47.180213 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:47.180384 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:47.181364 (+ 980us) server_negotiation.cc:299] Negotiation successful
0924 16:05:47.181668 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":270,"threads_started":1}
WARNING: no policy specified for kudu/127.3.214.195@KRBTEST.COM; defaulting to no policy
I20260924 16:05:47.183117 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:47.159009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39915 (local address 127.3.214.194:50815)
0924 16:05:47.159936 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:47.160003 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:47.162518 (+ 2515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:47.163459 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:47.163502 (+ 43us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:47.164422 (+ 920us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:47.165495 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:47.165524 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:47.166808 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:47.166814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:47.168815 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:47.168831 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:47.169834 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:47.169841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:47.170216 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:47.171026 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:47.171056 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:47.174210 (+ 3154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:47.177909 (+ 3699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:47.177927 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:47.177952 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:47.178805 (+ 853us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:47.179351 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:47.179355 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:47.179358 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:47.179613 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:47.180399 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:47.180407 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:47.180870 (+ 463us) client_negotiation.cc:769] Sending connection context
0924 16:05:47.181180 (+ 310us) client_negotiation.cc:240] Negotiation successful
0924 16:05:47.181689 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":329,"threads_started":1}
I20260924 16:05:47.184877 6772 heartbeater.cc:347] Connected to a master server at 127.3.214.254:39915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.195@KRBTEST.COM" created.
I20260924 16:05:47.185328 6772 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:47.186568 6772 heartbeater.cc:511] Master 127.3.214.254:39915 requested a full tablet report, sending...
I20260924 16:05:47.188911 6438 ts_manager.cc:194] Registered new tserver with Master: 546f475a10d046dd8fb89f551689644e (127.3.214.194:46643)
I20260924 16:05:47.189986 6438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:50815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
I20260924 16:05:47.213304 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39915
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:05:47.536530 6635 heartbeater.cc:503] Master 127.3.214.254:39915 was elected leader, sending a full tablet report...
W20260924 16:05:47.602660 6778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:47.603053 6778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:47.603204 6778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:47.611330 6778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:47.611474 6778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:47.611651 6778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:05:47.622846 6778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:47.624661 6778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:47.626408 6778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:47.641692 6785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:47.641741 6787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:47.641710 6784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:47.643178 6778 server_base.cc:1034] running on GCE node
I20260924 16:05:47.644014 6778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:47.645232 6778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:47.646469 6778 hybrid_clock.cc:648] HybridClock initialized: now 1790265947646399 us; error 51 us; skew 500 ppm
I20260924 16:05:47.646893 6778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:47 dist-test-slave-1gc3 krb5kdc[6390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265947, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:47.650701 6778 init.cc:377] Logged in from keytab as kudu/127.3.214.195@KRBTEST.COM (short username kudu)
I20260924 16:05:47.652545 6778 webserver.cc:460] Webserver started at http://127.3.214.195:40841/ using document root <none> and password file <none>
I20260924 16:05:47.653471 6778 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:47.653589 6778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:47.653924 6778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:47.657959 6778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "c1ca2165a2804c9c82c80e504d6055b0"
format_stamp: "Formatted at 2026-09-24 16:05:47 on dist-test-slave-1gc3"
server_key: "572984d9b003abc595db9482c46dbd34"
server_key_iv: "6fda12337e1a30dfb6fb7475a2d956f0"
server_key_version: "encryptionkey@0"
I20260924 16:05:47.659026 6778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "c1ca2165a2804c9c82c80e504d6055b0"
format_stamp: "Formatted at 2026-09-24 16:05:47 on dist-test-slave-1gc3"
server_key: "572984d9b003abc595db9482c46dbd34"
server_key_iv: "6fda12337e1a30dfb6fb7475a2d956f0"
server_key_version: "encryptionkey@0"
I20260924 16:05:47.666288 6778 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20260924 16:05:47.671325 6794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:47.673382 6778 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260924 16:05:47.673614 6778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "c1ca2165a2804c9c82c80e504d6055b0"
format_stamp: "Formatted at 2026-09-24 16:05:47 on dist-test-slave-1gc3"
server_key: "572984d9b003abc595db9482c46dbd34"
server_key_iv: "6fda12337e1a30dfb6fb7475a2d956f0"
server_key_version: "encryptionkey@0"
I20260924 16:05:47.673868 6778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:47.695586 6778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:47.700765 6778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:47.701108 6778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:47.702418 6778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:47.704169 6778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:47.704253 6778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:47.704365 6778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:47.704412 6778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:47.743713 6778 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:40913
I20260924 16:05:47.743767 6908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:40913 every 8 connection(s)
I20260924 16:05:47.745333 6778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:05:47.748389 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 6778
I20260924 16:05:47.748602 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:05:47.749222 3931 external_mini_cluster.cc:1428] Setting key 7d03aef39a2981efbff1bea8ee47971e
Sep 24 16:05:47 dist-test-slave-1gc3 krb5kdc[6390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265947, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:47.769455 6911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:47.748507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:42705 (local address 127.3.214.254:39915)
0924 16:05:47.749292 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:47.749300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:47.750419 (+ 1119us) server_negotiation.cc:407] Connection header received
0924 16:05:47.753272 (+ 2853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:47.753277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:47.753391 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:47.753498 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:47.755355 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:47.756158 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:47.757824 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:47.758295 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:47.758411 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:47.762550 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:47.762556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:47.762560 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:47.762601 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:47.765679 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:47.766846 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:47.766852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:47.766856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:47.766995 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:47.767456 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:47.767459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:47.767462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:47.767764 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:47.767921 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:47.768767 (+ 846us) server_negotiation.cc:299] Negotiation successful
0924 16:05:47.769028 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":312,"threads_started":1}
I20260924 16:05:47.770538 6912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:47.749172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39915 (local address 127.3.214.195:42705)
0924 16:05:47.750200 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:47.750265 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:47.752889 (+ 2624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:47.753776 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:47.753785 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:47.754481 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:47.755159 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:47.755182 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:47.756329 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:47.756333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:47.757668 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:47.757682 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:47.758560 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:47.758569 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:47.758817 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:47.759510 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:47.759541 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:47.762219 (+ 2678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:47.765865 (+ 3646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:47.765877 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:47.765895 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:47.766705 (+ 810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:47.767106 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:47.767111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:47.767115 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:47.767328 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:47.767897 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:47.767918 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:47.768351 (+ 433us) client_negotiation.cc:769] Sending connection context
0924 16:05:47.768617 (+ 266us) client_negotiation.cc:240] Negotiation successful
0924 16:05:47.769210 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":394,"threads_started":1}
I20260924 16:05:47.772168 6909 heartbeater.cc:347] Connected to a master server at 127.3.214.254:39915
I20260924 16:05:47.772668 6909 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:47.773764 6909 heartbeater.cc:511] Master 127.3.214.254:39915 requested a full tablet report, sending...
I20260924 16:05:47.775792 6438 ts_manager.cc:194] Registered new tserver with Master: c1ca2165a2804c9c82c80e504d6055b0 (127.3.214.195:40913)
I20260924 16:05:47.776731 6438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:42705
I20260924 16:05:47.779841 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 16:05:47.820880 6911 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:05:47.817584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40070 (local address 127.3.214.254:39915)
0924 16:05:47.817928 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:47.817935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:47.818358 (+ 423us) server_negotiation.cc:407] Connection header received
0924 16:05:47.818577 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:47.818582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:47.818674 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:47.818761 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:47.820585 (+ 1824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40070: BlockingRecv error: recv got EOF from 127.0.0.1:40070 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
I20260924 16:05:47.825577 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 6502
W20260924 16:05:47.828873 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 6502: Not authorized: Client connection negotiation failed: client connection to 127.3.214.193:46829: server requires authentication, but client does not have Kerberos credentials available
I20260924 16:05:47.829142 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 6502
I20260924 16:05:47.845724 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 6641
W20260924 16:05:47.848912 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 6641: Not authorized: Client connection negotiation failed: client connection to 127.3.214.194:46643: server requires authentication, but client does not have Kerberos credentials available
I20260924 16:05:47.849102 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 6641
I20260924 16:05:47.867678 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 6778
W20260924 16:05:47.871616 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 6778: Not authorized: Client connection negotiation failed: client connection to 127.3.214.195:40913: server requires authentication, but client does not have Kerberos credentials available
I20260924 16:05:47.871804 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 6778
I20260924 16:05:47.887812 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 6405
I20260924 16:05:47.889375 6469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:48.000797 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 6405
2026-09-24T16:05:48Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4020 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:05:48 dist-test-slave-1gc3 krb5kdc[6927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:05:48 dist-test-slave-1gc3 krb5kdc[6927](info): set up 2 sockets
Sep 24 16:05:48 dist-test-slave-1gc3 krb5kdc[6927](info): commencing operation
krb5kdc: starting...
W20260924 16:05:49.038012 3931 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:05:49 dist-test-slave-1gc3 krb5kdc[6927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:05:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:05:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:05:49.265857 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:42439
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:45087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:42439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:49.649003 6941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:49.649421 6941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:49.649554 6941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:49.657754 6941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:05:49.657910 6941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:49.657959 6941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:49.658051 6941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:05:49.658147 6941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:05:49.669780 6941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:45087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:42439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:42439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:49.671574 6941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:49.673287 6941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:49.687501 6947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:49.687489 6946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:49.687585 6949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:49.688742 6941 server_base.cc:1034] running on GCE node
I20260924 16:05:49.689795 6941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:49.691768 6941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:49.693023 6941 hybrid_clock.cc:648] HybridClock initialized: now 1790265949692962 us; error 51 us; skew 500 ppm
I20260924 16:05:49.693498 6941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:49 dist-test-slave-1gc3 krb5kdc[6927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265949, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:49.704278 6941 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:05:49.706578 6941 webserver.cc:460] Webserver started at http://127.3.214.254:43039/ using document root <none> and password file <none>
I20260924 16:05:49.707549 6941 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:49.707803 6941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:49.708254 6941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:49.712370 6941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "e52903705b6343048c7212fc4971a3d8"
format_stamp: "Formatted at 2026-09-24 16:05:49 on dist-test-slave-1gc3"
server_key: "3dcb2c2904b8c27fb490fd472a0d8f6c"
server_key_iv: "e768729a79c499ab4b91f2183b96c748"
server_key_version: "encryptionkey@0"
I20260924 16:05:49.713450 6941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "e52903705b6343048c7212fc4971a3d8"
format_stamp: "Formatted at 2026-09-24 16:05:49 on dist-test-slave-1gc3"
server_key: "3dcb2c2904b8c27fb490fd472a0d8f6c"
server_key_iv: "e768729a79c499ab4b91f2183b96c748"
server_key_version: "encryptionkey@0"
I20260924 16:05:49.721328 6941 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260924 16:05:49.726569 6956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:49.728678 6941 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260924 16:05:49.728962 6941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "e52903705b6343048c7212fc4971a3d8"
format_stamp: "Formatted at 2026-09-24 16:05:49 on dist-test-slave-1gc3"
server_key: "3dcb2c2904b8c27fb490fd472a0d8f6c"
server_key_iv: "e768729a79c499ab4b91f2183b96c748"
server_key_version: "encryptionkey@0"
I20260924 16:05:49.729230 6941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:49.765105 6941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:49.770555 6941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:49.770922 6941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:49.793869 6941 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:42439
I20260924 16:05:49.793908 7008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:42439 every 8 connection(s)
I20260924 16:05:49.795800 6941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:05:49.800271 7009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:49.801980 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 6941
I20260924 16:05:49.802253 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:05:49.803083 3931 external_mini_cluster.cc:1428] Setting key 17e106032e92e8559ebad76d0027a546
I20260924 16:05:49.815372 7009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e52903705b6343048c7212fc4971a3d8: Bootstrap starting.
I20260924 16:05:49.819717 7009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e52903705b6343048c7212fc4971a3d8: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:49.821086 7009 log.cc:824] T 00000000000000000000000000000000 P e52903705b6343048c7212fc4971a3d8: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:05:49 dist-test-slave-1gc3 krb5kdc[6927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:49.824333 7009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e52903705b6343048c7212fc4971a3d8: No bootstrap required, opened a new log
I20260924 16:05:49.833796 7012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:49.805837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34290 (local address 127.3.214.254:42439)
0924 16:05:49.807145 (+ 1308us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:49.807161 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:49.807227 (+ 66us) server_negotiation.cc:407] Connection header received
0924 16:05:49.809391 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:49.809426 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:49.810034 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:49.810581 (+ 547us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:49.812865 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:49.814500 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:49.816406 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:49.817383 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:49.817603 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:49.822803 (+ 5200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:49.822815 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:49.822842 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:49.822903 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:49.826632 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:49.827644 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:49.827654 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:49.827665 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:49.827906 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:49.828342 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:49.828346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:49.828349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:49.829471 (+ 1122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:49.829861 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:49.831530 (+ 1669us) server_negotiation.cc:299] Negotiation successful
0924 16:05:49.832182 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":993,"thread_start_us":609,"threads_started":1}
I20260924 16:05:49.834635 7009 raft_consensus.cc:348] T 00000000000000000000000000000000 P e52903705b6343048c7212fc4971a3d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52903705b6343048c7212fc4971a3d8" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42439 } }
I20260924 16:05:49.835058 7009 raft_consensus.cc:374] T 00000000000000000000000000000000 P e52903705b6343048c7212fc4971a3d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:49.835183 7009 raft_consensus.cc:729] T 00000000000000000000000000000000 P e52903705b6343048c7212fc4971a3d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e52903705b6343048c7212fc4971a3d8, State: Initialized, Role: FOLLOWER
I20260924 16:05:49.835906 7009 consensus_queue.cc:260] T 00000000000000000000000000000000 P e52903705b6343048c7212fc4971a3d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52903705b6343048c7212fc4971a3d8" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42439 } }
I20260924 16:05:49.836156 7009 raft_consensus.cc:388] T 00000000000000000000000000000000 P e52903705b6343048c7212fc4971a3d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:05:49.836313 7009 raft_consensus.cc:482] T 00000000000000000000000000000000 P e52903705b6343048c7212fc4971a3d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:05:49.836494 7009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e52903705b6343048c7212fc4971a3d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:49.839279 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:05:49.839224 7009 raft_consensus.cc:504] T 00000000000000000000000000000000 P e52903705b6343048c7212fc4971a3d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52903705b6343048c7212fc4971a3d8" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42439 } }
I20260924 16:05:49.840164 7009 leader_election.cc:302] T 00000000000000000000000000000000 P e52903705b6343048c7212fc4971a3d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e52903705b6343048c7212fc4971a3d8; no voters:
I20260924 16:05:49.840966 7009 leader_election.cc:288] T 00000000000000000000000000000000 P e52903705b6343048c7212fc4971a3d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:05:49.841145 7014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e52903705b6343048c7212fc4971a3d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:49.842437 7014 raft_consensus.cc:686] T 00000000000000000000000000000000 P e52903705b6343048c7212fc4971a3d8 [term 1 LEADER]: Becoming Leader. State: Replica: e52903705b6343048c7212fc4971a3d8, State: Running, Role: LEADER
I20260924 16:05:49.843103 7014 consensus_queue.cc:237] T 00000000000000000000000000000000 P e52903705b6343048c7212fc4971a3d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52903705b6343048c7212fc4971a3d8" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42439 } }
I20260924 16:05:49.843627 7009 sys_catalog.cc:564] T 00000000000000000000000000000000 P e52903705b6343048c7212fc4971a3d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:05:49.848294 7015 sys_catalog.cc:455] T 00000000000000000000000000000000 P e52903705b6343048c7212fc4971a3d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e52903705b6343048c7212fc4971a3d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52903705b6343048c7212fc4971a3d8" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42439 } } }
I20260924 16:05:49.848707 7015 sys_catalog.cc:458] T 00000000000000000000000000000000 P e52903705b6343048c7212fc4971a3d8 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:49.850318 7016 sys_catalog.cc:455] T 00000000000000000000000000000000 P e52903705b6343048c7212fc4971a3d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e52903705b6343048c7212fc4971a3d8. Latest consensus state: current_term: 1 leader_uuid: "e52903705b6343048c7212fc4971a3d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52903705b6343048c7212fc4971a3d8" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42439 } } }
I20260924 16:05:49.850875 7016 sys_catalog.cc:458] T 00000000000000000000000000000000 P e52903705b6343048c7212fc4971a3d8 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:49.852365 7019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:05:49.859556 7019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:05:49.878374 7019 catalog_manager.cc:1347] Generated new cluster ID: b5d9c00b13ff47948fc1cc27fd799490
I20260924 16:05:49.878525 7019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:05:49.897576 7019 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:05:49.900275 7019 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:05:49.912827 7019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e52903705b6343048c7212fc4971a3d8: Generated new TSK 0
I20260924 16:05:49.914410 7019 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:05:49.915570 7012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:49.904646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34300 (local address 127.3.214.254:42439)
0924 16:05:49.904872 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:49.904880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:49.905682 (+ 802us) server_negotiation.cc:407] Connection header received
0924 16:05:49.905922 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:49.905929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:49.906073 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:49.906241 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:49.907734 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:49.908977 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:49.910773 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:49.911430 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:49.911597 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:49.913366 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:49.913373 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:49.913378 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:05:49.914495 (+ 1117us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:49.914604 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:05:49.914651 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:49.914786 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:49.914985 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 16:05:49.915143 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":75}
I20260924 16:05:49.924995 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 6941
I20260924 16:05:49.926865 7004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:50.024027 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 6941
2026-09-24T16:05:50Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2016 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:05:50 dist-test-slave-1gc3 krb5kdc[7043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:05:50 dist-test-slave-1gc3 krb5kdc[7043](info): set up 2 sockets
Sep 24 16:05:50 dist-test-slave-1gc3 krb5kdc[7043](info): commencing operation
krb5kdc: starting...
W20260924 16:05:51.069077 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:05:51 dist-test-slave-1gc3 krb5kdc[7043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:05:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:05:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:05:51.306237 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:33163
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:33163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:51.693389 7056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:51.693861 7056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:51.694000 7056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:51.703238 7056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:05:51.703451 7056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:51.703531 7056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:51.703615 7056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:05:51.703681 7056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:05:51.716343 7056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:33163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:33163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:51.718408 7056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:51.720427 7056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:51.735878 7061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:51.735914 7062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:51.735965 7064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:51.737037 7056 server_base.cc:1034] running on GCE node
I20260924 16:05:51.738219 7056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:51.740216 7056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:51.741465 7056 hybrid_clock.cc:648] HybridClock initialized: now 1790265951741399 us; error 57 us; skew 500 ppm
I20260924 16:05:51.741925 7056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:51 dist-test-slave-1gc3 krb5kdc[7043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265951, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:51.745831 7056 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:05:51.748051 7056 webserver.cc:460] Webserver started at http://127.3.214.254:43005/ using document root <none> and password file <none>
I20260924 16:05:51.748960 7056 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:51.749106 7056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:51.749490 7056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:51.753335 7056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "a9f9807dff7c417da08bb244dc7016e1"
format_stamp: "Formatted at 2026-09-24 16:05:51 on dist-test-slave-1gc3"
server_key: "06c63c7c8b5696a86fa4d511691eefbd"
server_key_iv: "c686089896e6cb647526220a806849cf"
server_key_version: "encryptionkey@0"
I20260924 16:05:51.754395 7056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "a9f9807dff7c417da08bb244dc7016e1"
format_stamp: "Formatted at 2026-09-24 16:05:51 on dist-test-slave-1gc3"
server_key: "06c63c7c8b5696a86fa4d511691eefbd"
server_key_iv: "c686089896e6cb647526220a806849cf"
server_key_version: "encryptionkey@0"
I20260924 16:05:51.761992 7056 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260924 16:05:51.767092 7071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:51.769140 7056 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:05:51.769420 7056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "a9f9807dff7c417da08bb244dc7016e1"
format_stamp: "Formatted at 2026-09-24 16:05:51 on dist-test-slave-1gc3"
server_key: "06c63c7c8b5696a86fa4d511691eefbd"
server_key_iv: "c686089896e6cb647526220a806849cf"
server_key_version: "encryptionkey@0"
I20260924 16:05:51.769695 7056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:51.784935 7056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:51.790804 7056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:51.791196 7056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:51.814361 7056 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:33163
I20260924 16:05:51.814409 7124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:33163 every 8 connection(s)
I20260924 16:05:51.816265 7056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:05:51.821292 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:51.823657 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 7056
I20260924 16:05:51.823895 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:05:51.824599 3931 external_mini_cluster.cc:1428] Setting key 2cec1656a17cbc82458eff3b4334c597
I20260924 16:05:51.835497 7125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f9807dff7c417da08bb244dc7016e1: Bootstrap starting.
I20260924 16:05:51.839977 7125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9f9807dff7c417da08bb244dc7016e1: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:51.841521 7125 log.cc:824] T 00000000000000000000000000000000 P a9f9807dff7c417da08bb244dc7016e1: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:05:51 dist-test-slave-1gc3 krb5kdc[7043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:51.845535 7125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f9807dff7c417da08bb244dc7016e1: No bootstrap required, opened a new log
I20260924 16:05:51.853837 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:51.827525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.3.214.254:33163)
0924 16:05:51.828556 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:51.828572 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:51.828627 (+ 55us) server_negotiation.cc:407] Connection header received
0924 16:05:51.830773 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:51.830806 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:51.831393 (+ 587us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:51.831857 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:51.833849 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:51.835484 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:51.837432 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:51.838545 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:51.838762 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:51.844422 (+ 5660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:51.844434 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:51.844469 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:51.844518 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:51.848494 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:51.849451 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:51.849460 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:51.849470 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:51.849704 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:51.850153 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:51.850157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:51.850163 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:51.850931 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:51.851304 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:51.851832 (+ 528us) server_negotiation.cc:299] Negotiation successful
0924 16:05:51.852418 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":486,"threads_started":1}
I20260924 16:05:51.854323 7125 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9f9807dff7c417da08bb244dc7016e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f9807dff7c417da08bb244dc7016e1" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 33163 } }
I20260924 16:05:51.854903 7125 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9f9807dff7c417da08bb244dc7016e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:51.855019 7125 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9f9807dff7c417da08bb244dc7016e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f9807dff7c417da08bb244dc7016e1, State: Initialized, Role: FOLLOWER
I20260924 16:05:51.855700 7125 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9f9807dff7c417da08bb244dc7016e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f9807dff7c417da08bb244dc7016e1" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 33163 } }
I20260924 16:05:51.855938 7125 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9f9807dff7c417da08bb244dc7016e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:05:51.856081 7125 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9f9807dff7c417da08bb244dc7016e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:05:51.856240 7125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9f9807dff7c417da08bb244dc7016e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:51.859025 7125 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9f9807dff7c417da08bb244dc7016e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f9807dff7c417da08bb244dc7016e1" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 33163 } }
I20260924 16:05:51.859592 7125 leader_election.cc:302] T 00000000000000000000000000000000 P a9f9807dff7c417da08bb244dc7016e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9f9807dff7c417da08bb244dc7016e1; no voters:
I20260924 16:05:51.860435 7125 leader_election.cc:288] T 00000000000000000000000000000000 P a9f9807dff7c417da08bb244dc7016e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:05:51.860641 7130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9f9807dff7c417da08bb244dc7016e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:51.861583 7130 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9f9807dff7c417da08bb244dc7016e1 [term 1 LEADER]: Becoming Leader. State: Replica: a9f9807dff7c417da08bb244dc7016e1, State: Running, Role: LEADER
I20260924 16:05:51.862534 7130 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9f9807dff7c417da08bb244dc7016e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f9807dff7c417da08bb244dc7016e1" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 33163 } }
I20260924 16:05:51.863065 7125 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9f9807dff7c417da08bb244dc7016e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:05:51.866863 7131 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f9807dff7c417da08bb244dc7016e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9f9807dff7c417da08bb244dc7016e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f9807dff7c417da08bb244dc7016e1" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 33163 } } }
I20260924 16:05:51.866797 7132 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f9807dff7c417da08bb244dc7016e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9f9807dff7c417da08bb244dc7016e1. Latest consensus state: current_term: 1 leader_uuid: "a9f9807dff7c417da08bb244dc7016e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f9807dff7c417da08bb244dc7016e1" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 33163 } } }
I20260924 16:05:51.867473 7131 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f9807dff7c417da08bb244dc7016e1 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:51.867528 7132 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f9807dff7c417da08bb244dc7016e1 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:51.869532 7140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:05:51.876241 7140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:05:51.888433 7140 catalog_manager.cc:1347] Generated new cluster ID: 0e9f1d5fd7cd4f1eb94189968b7b7f0c
I20260924 16:05:51.888581 7140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:05:51.905642 7140 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:05:51.907184 7140 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:05:51.916306 7140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9f9807dff7c417da08bb244dc7016e1: Generated new TSK 0
I20260924 16:05:51.917932 7140 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.214.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
I20260924 16:05:51.975402 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:33163
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:52.340054 7152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:52.340373 7152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:52.340513 7152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:52.348287 7152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:52.348426 7152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:52.348594 7152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:05:52.359082 7152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:33163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:52.360602 7152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:52.362095 7152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:52.377890 7158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:52.378353 7157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:52.378541 7160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:52.378652 7152 server_base.cc:1034] running on GCE node
I20260924 16:05:52.379480 7152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:52.380719 7152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:52.381979 7152 hybrid_clock.cc:648] HybridClock initialized: now 1790265952381912 us; error 51 us; skew 500 ppm
I20260924 16:05:52.382396 7152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:52 dist-test-slave-1gc3 krb5kdc[7043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265952, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:52.386076 7152 init.cc:377] Logged in from keytab as kudu/127.3.214.193@KRBTEST.COM (short username kudu)
I20260924 16:05:52.387939 7152 webserver.cc:460] Webserver started at http://127.3.214.193:45625/ using document root <none> and password file <none>
I20260924 16:05:52.388810 7152 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:52.388940 7152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:52.389287 7152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:52.392952 7152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "6e1c50f64b074f33b71493600222f5b3"
format_stamp: "Formatted at 2026-09-24 16:05:52 on dist-test-slave-1gc3"
server_key: "21c56587a7df2dcaecf4b09f43c20b11"
server_key_iv: "d5d951cb3542f6df82ea84dcd98433ad"
server_key_version: "encryptionkey@0"
I20260924 16:05:52.393951 7152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "6e1c50f64b074f33b71493600222f5b3"
format_stamp: "Formatted at 2026-09-24 16:05:52 on dist-test-slave-1gc3"
server_key: "21c56587a7df2dcaecf4b09f43c20b11"
server_key_iv: "d5d951cb3542f6df82ea84dcd98433ad"
server_key_version: "encryptionkey@0"
I20260924 16:05:52.400733 7152 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 16:05:52.405135 7167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:52.407280 7152 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:05:52.407531 7152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "6e1c50f64b074f33b71493600222f5b3"
format_stamp: "Formatted at 2026-09-24 16:05:52 on dist-test-slave-1gc3"
server_key: "21c56587a7df2dcaecf4b09f43c20b11"
server_key_iv: "d5d951cb3542f6df82ea84dcd98433ad"
server_key_version: "encryptionkey@0"
I20260924 16:05:52.407799 7152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:52.456084 7152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:52.461779 7152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:52.462234 7152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:52.463599 7152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:52.465504 7152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:52.465615 7152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:52.465767 7152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:52.465868 7152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:52.505515 7152 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:33725
I20260924 16:05:52.505568 7281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:33725 every 8 connection(s)
I20260924 16:05:52.507313 7152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:05:52.513840 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 7152
I20260924 16:05:52.514031 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:05:52.514732 3931 external_mini_cluster.cc:1428] Setting key 0bef4fad8df507e0c6de9ab569e8213b
Sep 24 16:05:52 dist-test-slave-1gc3 krb5kdc[7043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265952, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:52.531520 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:52.510320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:43561 (local address 127.3.214.254:33163)
0924 16:05:52.510898 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:52.510903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:52.512010 (+ 1107us) server_negotiation.cc:407] Connection header received
0924 16:05:52.514476 (+ 2466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:52.514482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:52.514592 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:52.514703 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:52.516891 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:52.517696 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:52.519546 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:52.519991 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:52.520127 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:52.524788 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:52.524794 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:52.524798 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:52.524844 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:52.527765 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:52.528710 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:52.528715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:52.528718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:52.528856 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:52.529348 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:52.529352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:52.529354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:52.529687 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:52.529835 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:52.530816 (+ 981us) server_negotiation.cc:299] Negotiation successful
0924 16:05:52.531104 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":283,"threads_started":1}
WARNING: no policy specified for kudu/127.3.214.194@KRBTEST.COM; defaulting to no policy
I20260924 16:05:52.532802 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:52.510859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:33163 (local address 127.3.214.193:43561)
0924 16:05:52.511785 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:52.511840 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:52.514084 (+ 2244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:52.515025 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:52.515036 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:52.515704 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:52.516694 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:52.516722 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:52.517877 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:52.517881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:52.519377 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:52.519393 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:52.520257 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:52.520266 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:52.520560 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:52.521345 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:52.521373 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:52.524362 (+ 2989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:52.527965 (+ 3603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:52.527978 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:52.527999 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:52.528557 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:52.528965 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:52.528971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:52.528976 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:52.529222 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:52.529821 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:52.529830 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:52.530335 (+ 505us) client_negotiation.cc:769] Sending connection context
0924 16:05:52.530618 (+ 283us) client_negotiation.cc:240] Negotiation successful
0924 16:05:52.531345 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":354,"threads_started":1}
I20260924 16:05:52.534937 7282 heartbeater.cc:347] Connected to a master server at 127.3.214.254:33163
I20260924 16:05:52.535319 7282 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.194@KRBTEST.COM" created.
I20260924 16:05:52.536293 7282 heartbeater.cc:511] Master 127.3.214.254:33163 requested a full tablet report, sending...
I20260924 16:05:52.539376 7089 ts_manager.cc:194] Registered new tserver with Master: 6e1c50f64b074f33b71493600222f5b3 (127.3.214.193:33725)
I20260924 16:05:52.541352 7089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:43561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
I20260924 16:05:52.561614 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:33163
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:52.962615 7289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:52.963033 7289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:52.963181 7289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:52.971740 7289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:52.971931 7289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:52.972126 7289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:05:52.983791 7289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:33163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:52.985720 7289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:52.987512 7289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:53.004257 7297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:53.004586 7294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:53.004618 7295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:53.006553 7289 server_base.cc:1034] running on GCE node
I20260924 16:05:53.007536 7289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:53.008961 7289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:53.010296 7289 hybrid_clock.cc:648] HybridClock initialized: now 1790265953010220 us; error 72 us; skew 500 ppm
I20260924 16:05:53.010747 7289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:53 dist-test-slave-1gc3 krb5kdc[7043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265953, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:53.015285 7289 init.cc:377] Logged in from keytab as kudu/127.3.214.194@KRBTEST.COM (short username kudu)
I20260924 16:05:53.017277 7289 webserver.cc:460] Webserver started at http://127.3.214.194:33243/ using document root <none> and password file <none>
I20260924 16:05:53.018181 7289 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:53.018316 7289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:53.018692 7289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:53.022215 7289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "2456fec33b5d4938bb84bb31bb22cd2c"
format_stamp: "Formatted at 2026-09-24 16:05:53 on dist-test-slave-1gc3"
server_key: "6f0212da1e169aa73ac5a5ba47d39209"
server_key_iv: "0163a601b453b2df117c302b12ec0fe8"
server_key_version: "encryptionkey@0"
I20260924 16:05:53.023269 7289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "2456fec33b5d4938bb84bb31bb22cd2c"
format_stamp: "Formatted at 2026-09-24 16:05:53 on dist-test-slave-1gc3"
server_key: "6f0212da1e169aa73ac5a5ba47d39209"
server_key_iv: "0163a601b453b2df117c302b12ec0fe8"
server_key_version: "encryptionkey@0"
I20260924 16:05:53.029929 7289 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 16:05:53.034778 7304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:53.037052 7289 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:05:53.037321 7289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "2456fec33b5d4938bb84bb31bb22cd2c"
format_stamp: "Formatted at 2026-09-24 16:05:53 on dist-test-slave-1gc3"
server_key: "6f0212da1e169aa73ac5a5ba47d39209"
server_key_iv: "0163a601b453b2df117c302b12ec0fe8"
server_key_version: "encryptionkey@0"
I20260924 16:05:53.037590 7289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:53.062260 7289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:53.068372 7289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:53.068719 7289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:53.069971 7289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:53.071669 7289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:53.071763 7289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:53.071873 7289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:53.071942 7289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:53.109872 7289 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:41831
I20260924 16:05:53.109980 7418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:41831 every 8 connection(s)
I20260924 16:05:53.111351 7289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:05:53.117336 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 7289
I20260924 16:05:53.117512 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:05:53.117969 3931 external_mini_cluster.cc:1428] Setting key 452838f0343cb08d10ef8f906df9b823
Sep 24 16:05:53 dist-test-slave-1gc3 krb5kdc[7043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265953, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:53.139371 7421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:53.114879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:38181 (local address 127.3.214.254:33163)
0924 16:05:53.115507 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:53.115513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:53.116707 (+ 1194us) server_negotiation.cc:407] Connection header received
0924 16:05:53.119319 (+ 2612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:53.119327 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:53.119447 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:53.119562 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:53.121852 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:53.122762 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:53.125596 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:53.126261 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:53.126475 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:53.131482 (+ 5007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:53.131489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:53.131493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:53.131546 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:53.134521 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:53.135892 (+ 1371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:53.135898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:53.135901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:53.136090 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:53.136783 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:53.136788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:53.136790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:53.137238 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:53.137428 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:53.138810 (+ 1382us) server_negotiation.cc:299] Negotiation successful
0924 16:05:53.139066 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":313,"threads_started":1}
I20260924 16:05:53.141460 7422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:53.115443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:33163 (local address 127.3.214.194:38181)
0924 16:05:53.116449 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:53.116511 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:53.118865 (+ 2354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:53.119951 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:53.119964 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:53.120724 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:53.121574 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:53.121599 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:53.123138 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:53.123146 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:53.125263 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:53.125297 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:53.126610 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:53.126617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:53.126869 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:53.127710 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:53.127738 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:53.130958 (+ 3220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:53.134768 (+ 3810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:53.134782 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:53.134811 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:53.135671 (+ 860us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:53.136249 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:53.136254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:53.136258 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:53.136611 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:53.137430 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:53.137441 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:53.138152 (+ 711us) client_negotiation.cc:769] Sending connection context
0924 16:05:53.138683 (+ 531us) client_negotiation.cc:240] Negotiation successful
0924 16:05:53.139516 (+ 833us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":338,"threads_started":1}
WARNING: no policy specified for kudu/127.3.214.195@KRBTEST.COM; defaulting to no policy
I20260924 16:05:53.143733 7419 heartbeater.cc:347] Connected to a master server at 127.3.214.254:33163
I20260924 16:05:53.144155 7419 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.195@KRBTEST.COM" created.
I20260924 16:05:53.145190 7419 heartbeater.cc:511] Master 127.3.214.254:33163 requested a full tablet report, sending...
I20260924 16:05:53.147465 7089 ts_manager.cc:194] Registered new tserver with Master: 2456fec33b5d4938bb84bb31bb22cd2c (127.3.214.194:41831)
I20260924 16:05:53.148494 7089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:38181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
I20260924 16:05:53.170080 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:33163
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:05:53.545644 7282 heartbeater.cc:503] Master 127.3.214.254:33163 was elected leader, sending a full tablet report...
W20260924 16:05:53.581517 7425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:53.581907 7425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:53.582019 7425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:53.590197 7425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:53.590389 7425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:53.590570 7425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:05:53.602006 7425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:33163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:53.603927 7425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:53.605644 7425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:53.622237 7431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:53.622249 7433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:53.622431 7430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:53.623783 7425 server_base.cc:1034] running on GCE node
I20260924 16:05:53.624763 7425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:53.626148 7425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:53.627414 7425 hybrid_clock.cc:648] HybridClock initialized: now 1790265953627356 us; error 64 us; skew 500 ppm
I20260924 16:05:53.627804 7425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:53 dist-test-slave-1gc3 krb5kdc[7043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265953, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:53.631680 7425 init.cc:377] Logged in from keytab as kudu/127.3.214.195@KRBTEST.COM (short username kudu)
I20260924 16:05:53.633599 7425 webserver.cc:460] Webserver started at http://127.3.214.195:44771/ using document root <none> and password file <none>
I20260924 16:05:53.634529 7425 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:53.634666 7425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:53.635025 7425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:53.638727 7425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "510854b7fc2f4f789c0d1128ccd802c1"
format_stamp: "Formatted at 2026-09-24 16:05:53 on dist-test-slave-1gc3"
server_key: "803a3c52ac8bf36501f7fec16ca2751d"
server_key_iv: "56cb75720427f9c9e6dd60c86c82655c"
server_key_version: "encryptionkey@0"
I20260924 16:05:53.639684 7425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "510854b7fc2f4f789c0d1128ccd802c1"
format_stamp: "Formatted at 2026-09-24 16:05:53 on dist-test-slave-1gc3"
server_key: "803a3c52ac8bf36501f7fec16ca2751d"
server_key_iv: "56cb75720427f9c9e6dd60c86c82655c"
server_key_version: "encryptionkey@0"
I20260924 16:05:53.646782 7425 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260924 16:05:53.651643 7440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:53.653625 7425 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:05:53.653869 7425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "510854b7fc2f4f789c0d1128ccd802c1"
format_stamp: "Formatted at 2026-09-24 16:05:53 on dist-test-slave-1gc3"
server_key: "803a3c52ac8bf36501f7fec16ca2751d"
server_key_iv: "56cb75720427f9c9e6dd60c86c82655c"
server_key_version: "encryptionkey@0"
I20260924 16:05:53.654138 7425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:53.681995 7425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:53.687582 7425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:53.687949 7425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:53.689229 7425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:53.691049 7425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:53.691143 7425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:53.691279 7425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:53.691334 7425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:53.731097 7425 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:40559
I20260924 16:05:53.731191 7556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:40559 every 8 connection(s)
I20260924 16:05:53.732728 7425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:05:53.735723 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 7425
I20260924 16:05:53.735940 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:05:53.736513 3931 external_mini_cluster.cc:1428] Setting key aa10167886a1d94f2bddd4eb46885f37
Sep 24 16:05:53 dist-test-slave-1gc3 krb5kdc[7043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265953, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:53.759415 7559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:53.736120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:60739 (local address 127.3.214.254:33163)
0924 16:05:53.736690 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:53.736698 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:53.738206 (+ 1508us) server_negotiation.cc:407] Connection header received
0924 16:05:53.741383 (+ 3177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:53.741388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:53.741498 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:53.741616 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:53.744033 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:53.744935 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:53.746918 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:53.747392 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:53.747515 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:53.752150 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:53.752157 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:53.752161 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:53.752213 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:53.755585 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:53.756633 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:53.756637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:53.756640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:53.756823 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:53.757362 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:53.757366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:53.757368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:53.757702 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:53.757869 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:53.758833 (+ 964us) server_negotiation.cc:299] Negotiation successful
0924 16:05:53.759070 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":273,"threads_started":1}
I20260924 16:05:53.760497 7560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:53.736775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:33163 (local address 127.3.214.195:60739)
0924 16:05:53.737937 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:53.738028 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:53.740998 (+ 2970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:53.741946 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:53.741959 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:53.742928 (+ 969us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:53.743798 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:53.743822 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:53.745115 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:53.745119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:53.746703 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:53.746730 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:53.747682 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:53.747688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:53.747962 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:53.748829 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:53.748858 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:53.751789 (+ 2931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:53.755815 (+ 4026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:53.755826 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:53.755847 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:53.756476 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:53.756971 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:53.756975 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:53.756977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:53.757214 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:53.757843 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:53.757850 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:53.758333 (+ 483us) client_negotiation.cc:769] Sending connection context
0924 16:05:53.758686 (+ 353us) client_negotiation.cc:240] Negotiation successful
0924 16:05:53.759169 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":448,"threads_started":1}
I20260924 16:05:53.762255 7557 heartbeater.cc:347] Connected to a master server at 127.3.214.254:33163
I20260924 16:05:53.762719 7557 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:53.763782 7557 heartbeater.cc:511] Master 127.3.214.254:33163 requested a full tablet report, sending...
I20260924 16:05:53.766037 7088 ts_manager.cc:194] Registered new tserver with Master: 510854b7fc2f4f789c0d1128ccd802c1 (127.3.214.195:40559)
I20260924 16:05:53.767378 7088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:60739
I20260924 16:05:53.767601 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 16:05:53 dist-test-slave-1gc3 krb5kdc[7043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265953, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 16:05:53 dist-test-slave-1gc3 krb5kdc[7043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265953, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:53.829471 7559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:53.810864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.3.214.254:33163)
0924 16:05:53.811093 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:53.811098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:53.811715 (+ 617us) server_negotiation.cc:407] Connection header received
0924 16:05:53.811905 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:53.811910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:53.811992 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:53.812171 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:53.813865 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:53.815153 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:53.816905 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:53.817576 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:53.817740 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:53.823139 (+ 5399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:53.823146 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:53.823150 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:53.823210 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:53.826290 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:53.827263 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:53.827268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:53.827270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:53.827447 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:53.827916 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:53.827919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:53.827921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:53.828340 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:53.828558 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:53.828941 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 16:05:53.829174 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260924 16:05:53.830212 7088 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56716
I20260924 16:05:53.833107 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 7152
Sep 24 16:05:53 dist-test-slave-1gc3 krb5kdc[7043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265953, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.214.193@KRBTEST.COM
I20260924 16:05:53.854241 7569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:53.834412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55602 (local address 127.3.214.193:33725)
0924 16:05:53.835220 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:53.835228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:53.835257 (+ 29us) server_negotiation.cc:407] Connection header received
0924 16:05:53.835329 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:53.835336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:53.835577 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:53.835739 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:53.837268 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:53.838424 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:53.840280 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:53.840962 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:53.841143 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:53.846094 (+ 4951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:53.846150 (+ 56us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:53.846177 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:53.846236 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:53.850794 (+ 4558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:53.851817 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:53.851826 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:53.851829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:53.852000 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:53.852467 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:53.852471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:53.852473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:53.852915 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:53.853201 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:53.853594 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 16:05:53.853855 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":361,"threads_started":1}
W20260924 16:05:53.856032 7277 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55602
W20260924 16:05:53.857477 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 7152: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:05:53.857788 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 7152
I20260924 16:05:53.875416 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 7289
Sep 24 16:05:53 dist-test-slave-1gc3 krb5kdc[7043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265953, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.214.194@KRBTEST.COM
I20260924 16:05:53.895313 7570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:53.876672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50664 (local address 127.3.214.194:41831)
0924 16:05:53.877271 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:53.877279 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:53.877311 (+ 32us) server_negotiation.cc:407] Connection header received
0924 16:05:53.877384 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:53.877391 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:53.877685 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:53.877857 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:53.879423 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:53.880736 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:53.882358 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:53.883035 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:53.883170 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:53.888165 (+ 4995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:53.888188 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:53.888210 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:53.888280 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:53.891497 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:53.892539 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:53.892557 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:53.892560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:53.892778 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:53.893210 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:53.893215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:53.893217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:53.893778 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:53.894073 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:53.894536 (+ 463us) server_negotiation.cc:299] Negotiation successful
0924 16:05:53.894941 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":296,"threads_started":1}
W20260924 16:05:53.896656 7414 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50664
W20260924 16:05:53.897943 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 7289: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:05:53.898217 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 7289
I20260924 16:05:53.916957 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 7425
Sep 24 16:05:53 dist-test-slave-1gc3 krb5kdc[7043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265953, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.214.195@KRBTEST.COM
I20260924 16:05:53.936991 7571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:53.918503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35188 (local address 127.3.214.195:40559)
0924 16:05:53.919443 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:53.919455 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:53.919492 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:05:53.919564 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:53.919571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:53.919806 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:53.919998 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:53.921544 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:53.922752 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:53.924052 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:53.924726 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:53.924855 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:53.930172 (+ 5317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:53.930193 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:53.930213 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:53.930262 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:53.933250 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:53.934517 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:53.934529 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:53.934532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:53.934731 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:53.935232 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:53.935236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:53.935238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:53.935641 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:53.935935 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:53.936310 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 16:05:53.936587 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":481,"threads_started":1}
W20260924 16:05:53.939004 7552 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35188
W20260924 16:05:53.940549 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 7425: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:05:53.940768 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 7425
I20260924 16:05:53.959512 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 7056
I20260924 16:05:53.961196 7120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:54.087072 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 7056
2026-09-24T16:05:54Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4066 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:05:54 dist-test-slave-1gc3 krb5kdc[7575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:05:54 dist-test-slave-1gc3 krb5kdc[7575](info): set up 2 sockets
Sep 24 16:05:54 dist-test-slave-1gc3 krb5kdc[7575](info): commencing operation
krb5kdc: starting...
W20260924 16:05:55.124003 3931 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:05:55 dist-test-slave-1gc3 krb5kdc[7575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:05:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:05:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:05:55.359814 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:38395
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:38395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:55.755378 7588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:55.755776 7588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:55.755906 7588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:55.764045 7588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:05:55.764196 7588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:55.764253 7588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:55.764344 7588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:05:55.764412 7588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:05:55.775694 7588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:38395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:38395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:55.777515 7588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:55.779183 7588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:55.793906 7595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:55.794067 7596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:55.794040 7598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:55.795099 7588 server_base.cc:1034] running on GCE node
I20260924 16:05:55.796211 7588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:55.798235 7588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:55.799499 7588 hybrid_clock.cc:648] HybridClock initialized: now 1790265955799437 us; error 56 us; skew 500 ppm
I20260924 16:05:55.799961 7588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:55 dist-test-slave-1gc3 krb5kdc[7575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265955, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:55.812633 7588 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:05:55.815043 7588 webserver.cc:460] Webserver started at http://127.3.214.254:44245/ using document root <none> and password file <none>
I20260924 16:05:55.816047 7588 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:55.816210 7588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:55.816630 7588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:55.820850 7588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "12e0c92a5dac40b6b927a58da9bc58f4"
format_stamp: "Formatted at 2026-09-24 16:05:55 on dist-test-slave-1gc3"
server_key: "a0682ed18052be394132769eddffd121"
server_key_iv: "e20498d1db4633488a04f72a6ab11c65"
server_key_version: "encryptionkey@0"
I20260924 16:05:55.821993 7588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "12e0c92a5dac40b6b927a58da9bc58f4"
format_stamp: "Formatted at 2026-09-24 16:05:55 on dist-test-slave-1gc3"
server_key: "a0682ed18052be394132769eddffd121"
server_key_iv: "e20498d1db4633488a04f72a6ab11c65"
server_key_version: "encryptionkey@0"
I20260924 16:05:55.829627 7588 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s
I20260924 16:05:55.835204 7606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:55.837661 7588 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260924 16:05:55.838011 7588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "12e0c92a5dac40b6b927a58da9bc58f4"
format_stamp: "Formatted at 2026-09-24 16:05:55 on dist-test-slave-1gc3"
server_key: "a0682ed18052be394132769eddffd121"
server_key_iv: "e20498d1db4633488a04f72a6ab11c65"
server_key_version: "encryptionkey@0"
I20260924 16:05:55.838367 7588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:55.862437 7588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:55.867772 7588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:55.868079 7588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:55.890172 7588 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:38395
I20260924 16:05:55.890199 7657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:38395 every 8 connection(s)
I20260924 16:05:55.891940 7588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:05:55.896565 7658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:05:55.896656 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 7588
I20260924 16:05:55.896888 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:05:55.897704 3931 external_mini_cluster.cc:1428] Setting key 8a4204fbaa7894136b185cb4f7d5fb0b
I20260924 16:05:55.912837 7658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12e0c92a5dac40b6b927a58da9bc58f4: Bootstrap starting.
Sep 24 16:05:55 dist-test-slave-1gc3 krb5kdc[7575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:55.917148 7658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12e0c92a5dac40b6b927a58da9bc58f4: Neither blocks nor log segments found. Creating new log.
I20260924 16:05:55.918484 7658 log.cc:824] T 00000000000000000000000000000000 P 12e0c92a5dac40b6b927a58da9bc58f4: Log is configured to *not* fsync() on all Append() calls
I20260924 16:05:55.922561 7658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12e0c92a5dac40b6b927a58da9bc58f4: No bootstrap required, opened a new log
I20260924 16:05:55.926653 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:55.900408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47100 (local address 127.3.214.254:38395)
0924 16:05:55.901438 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:55.901453 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:55.901509 (+ 56us) server_negotiation.cc:407] Connection header received
0924 16:05:55.903956 (+ 2447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:55.903991 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:55.904551 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:55.904998 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:55.907056 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:55.908550 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:55.910577 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:55.911416 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:55.911585 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:55.917035 (+ 5450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:55.917048 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:55.917073 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:55.917158 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:55.921243 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:55.922380 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:55.922388 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:55.922397 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:55.922607 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:55.923079 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:55.923082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:55.923084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:55.923829 (+ 745us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:55.924095 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:55.924665 (+ 570us) server_negotiation.cc:299] Negotiation successful
0924 16:05:55.925206 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":463,"threads_started":1}
I20260924 16:05:55.932233 7658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12e0c92a5dac40b6b927a58da9bc58f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e0c92a5dac40b6b927a58da9bc58f4" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38395 } }
I20260924 16:05:55.932675 7658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12e0c92a5dac40b6b927a58da9bc58f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:05:55.932780 7658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12e0c92a5dac40b6b927a58da9bc58f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e0c92a5dac40b6b927a58da9bc58f4, State: Initialized, Role: FOLLOWER
I20260924 16:05:55.933898 7658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12e0c92a5dac40b6b927a58da9bc58f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e0c92a5dac40b6b927a58da9bc58f4" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38395 } }
I20260924 16:05:55.934293 7658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12e0c92a5dac40b6b927a58da9bc58f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:05:55.934468 7658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12e0c92a5dac40b6b927a58da9bc58f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:05:55.934657 7658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12e0c92a5dac40b6b927a58da9bc58f4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:05:55.937090 7658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12e0c92a5dac40b6b927a58da9bc58f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e0c92a5dac40b6b927a58da9bc58f4" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38395 } }
I20260924 16:05:55.937757 7658 leader_election.cc:302] T 00000000000000000000000000000000 P 12e0c92a5dac40b6b927a58da9bc58f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12e0c92a5dac40b6b927a58da9bc58f4; no voters:
I20260924 16:05:55.938560 7658 leader_election.cc:288] T 00000000000000000000000000000000 P 12e0c92a5dac40b6b927a58da9bc58f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:05:55.938776 7663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12e0c92a5dac40b6b927a58da9bc58f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:05:55.940675 7658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12e0c92a5dac40b6b927a58da9bc58f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:05:55.941148 7663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12e0c92a5dac40b6b927a58da9bc58f4 [term 1 LEADER]: Becoming Leader. State: Replica: 12e0c92a5dac40b6b927a58da9bc58f4, State: Running, Role: LEADER
I20260924 16:05:55.941944 7663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12e0c92a5dac40b6b927a58da9bc58f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e0c92a5dac40b6b927a58da9bc58f4" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38395 } }
I20260924 16:05:55.946604 7664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12e0c92a5dac40b6b927a58da9bc58f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12e0c92a5dac40b6b927a58da9bc58f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e0c92a5dac40b6b927a58da9bc58f4" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38395 } } }
I20260924 16:05:55.946457 7665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12e0c92a5dac40b6b927a58da9bc58f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12e0c92a5dac40b6b927a58da9bc58f4. Latest consensus state: current_term: 1 leader_uuid: "12e0c92a5dac40b6b927a58da9bc58f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e0c92a5dac40b6b927a58da9bc58f4" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38395 } } }
I20260924 16:05:55.947190 7664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12e0c92a5dac40b6b927a58da9bc58f4 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:55.947319 7665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12e0c92a5dac40b6b927a58da9bc58f4 [sys.catalog]: This master's current role is: LEADER
I20260924 16:05:55.952752 7674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:05:55.959681 7674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:05:55.972009 7674 catalog_manager.cc:1347] Generated new cluster ID: 363295c3f1144b97a6fad9826db2675c
I20260924 16:05:55.972193 7674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:05:55.981299 7674 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:05:55.983026 7674 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:05:55.993400 7674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12e0c92a5dac40b6b927a58da9bc58f4: Generated new TSK 0
I20260924 16:05:55.994752 7674 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.214.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
I20260924 16:05:56.061842 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:38395
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:56.467722 7684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:56.468103 7684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:56.468211 7684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:56.476461 7684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:56.476630 7684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:56.476799 7684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:05:56.488044 7684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:38395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:56.489763 7684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:56.491452 7684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:56.507838 7692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:56.507838 7689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:56.508109 7690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:56.509341 7684 server_base.cc:1034] running on GCE node
I20260924 16:05:56.510272 7684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:56.511560 7684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:56.512813 7684 hybrid_clock.cc:648] HybridClock initialized: now 1790265956512750 us; error 54 us; skew 500 ppm
I20260924 16:05:56.513221 7684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:56 dist-test-slave-1gc3 krb5kdc[7575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265956, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:56.517182 7684 init.cc:377] Logged in from keytab as kudu/127.3.214.193@KRBTEST.COM (short username kudu)
I20260924 16:05:56.519188 7684 webserver.cc:460] Webserver started at http://127.3.214.193:35281/ using document root <none> and password file <none>
I20260924 16:05:56.520063 7684 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:56.520177 7684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:56.520532 7684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:56.524271 7684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "97a25e6723794eae956275a8b1ca1a13"
format_stamp: "Formatted at 2026-09-24 16:05:56 on dist-test-slave-1gc3"
server_key: "8290ff1f327a2022e88fd8403b7032d5"
server_key_iv: "10c3b7e233123b2a928e11a3b2d79bea"
server_key_version: "encryptionkey@0"
I20260924 16:05:56.525381 7684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "97a25e6723794eae956275a8b1ca1a13"
format_stamp: "Formatted at 2026-09-24 16:05:56 on dist-test-slave-1gc3"
server_key: "8290ff1f327a2022e88fd8403b7032d5"
server_key_iv: "10c3b7e233123b2a928e11a3b2d79bea"
server_key_version: "encryptionkey@0"
I20260924 16:05:56.532330 7684 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260924 16:05:56.537506 7699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:56.539656 7684 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260924 16:05:56.539897 7684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "97a25e6723794eae956275a8b1ca1a13"
format_stamp: "Formatted at 2026-09-24 16:05:56 on dist-test-slave-1gc3"
server_key: "8290ff1f327a2022e88fd8403b7032d5"
server_key_iv: "10c3b7e233123b2a928e11a3b2d79bea"
server_key_version: "encryptionkey@0"
I20260924 16:05:56.540172 7684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:56.564327 7684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:56.569934 7684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:56.570351 7684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:56.571707 7684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:56.573529 7684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:56.573624 7684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:56.573791 7684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:56.573848 7684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:56.614086 7684 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:46043
I20260924 16:05:56.614207 7814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:46043 every 8 connection(s)
I20260924 16:05:56.615849 7684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:05:56.620049 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 7684
I20260924 16:05:56.620272 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:05:56.620913 3931 external_mini_cluster.cc:1428] Setting key a8bad53518500a08c2a5f26a115a18ff
Sep 24 16:05:56 dist-test-slave-1gc3 krb5kdc[7575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265956, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:56.642896 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:56.619174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:42031 (local address 127.3.214.254:38395)
0924 16:05:56.619798 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:56.619806 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:56.622449 (+ 2643us) server_negotiation.cc:407] Connection header received
0924 16:05:56.624797 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:56.624803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:56.624911 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:56.625010 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:56.626922 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.627685 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.629358 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.629782 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.629903 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:56.634734 (+ 4831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:56.634742 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:56.634747 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:56.634804 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:56.638303 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:56.639510 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:56.639515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:56.639519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:56.639707 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:56.640358 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:56.640363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:56.640367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:56.640825 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:56.641081 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:56.642069 (+ 988us) server_negotiation.cc:299] Negotiation successful
0924 16:05:56.642440 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":287,"threads_started":1}
WARNING: no policy specified for kudu/127.3.214.194@KRBTEST.COM; defaulting to no policy
I20260924 16:05:56.644218 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:56.620004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38395 (local address 127.3.214.193:42031)
0924 16:05:56.622238 (+ 2234us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:56.622310 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:56.624467 (+ 2157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:56.625324 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:56.625334 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:56.625959 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:56.626717 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:56.626739 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.627869 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.627873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:56.629194 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:56.629209 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:56.630059 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:56.630066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:56.630359 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:56.631405 (+ 1046us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:56.631451 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:56.634325 (+ 2874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:56.638584 (+ 4259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:56.638597 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:56.638620 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:56.639312 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:56.639853 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:56.639857 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:56.639860 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:56.640196 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:56.640979 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:56.640987 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:56.641494 (+ 507us) client_negotiation.cc:769] Sending connection context
0924 16:05:56.641874 (+ 380us) client_negotiation.cc:240] Negotiation successful
0924 16:05:56.642519 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1778,"thread_start_us":1512,"threads_started":1}
I20260924 16:05:56.646555 7815 heartbeater.cc:347] Connected to a master server at 127.3.214.254:38395
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.194@KRBTEST.COM" created.
I20260924 16:05:56.646983 7815 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:56.647905 7815 heartbeater.cc:511] Master 127.3.214.254:38395 requested a full tablet report, sending...
I20260924 16:05:56.650393 7623 ts_manager.cc:194] Registered new tserver with Master: 97a25e6723794eae956275a8b1ca1a13 (127.3.214.193:46043)
I20260924 16:05:56.651996 7623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:42031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
I20260924 16:05:56.678165 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:38395
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:57.078701 7821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:57.079134 7821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:57.079275 7821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:05:57.087677 7821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:57.087841 7821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:57.088009 7821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:05:57.099397 7821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:38395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:57.101236 7821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:57.102988 7821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:57.119445 7826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:57.119485 7827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:57.119547 7829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:57.120959 7821 server_base.cc:1034] running on GCE node
I20260924 16:05:57.121814 7821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:57.123113 7821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:57.124358 7821 hybrid_clock.cc:648] HybridClock initialized: now 1790265957124259 us; error 88 us; skew 500 ppm
I20260924 16:05:57.124751 7821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:57 dist-test-slave-1gc3 krb5kdc[7575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265957, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:57.128741 7821 init.cc:377] Logged in from keytab as kudu/127.3.214.194@KRBTEST.COM (short username kudu)
I20260924 16:05:57.130776 7821 webserver.cc:460] Webserver started at http://127.3.214.194:39223/ using document root <none> and password file <none>
I20260924 16:05:57.131701 7821 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:57.131834 7821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:57.132234 7821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:57.135939 7821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "c2c01f5adef94944ab19f023ca2c858a"
format_stamp: "Formatted at 2026-09-24 16:05:57 on dist-test-slave-1gc3"
server_key: "a055a80d6d8d8e167582dfc861888275"
server_key_iv: "2ce34534e02e37c88b9863bc54f0c43f"
server_key_version: "encryptionkey@0"
I20260924 16:05:57.136883 7821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "c2c01f5adef94944ab19f023ca2c858a"
format_stamp: "Formatted at 2026-09-24 16:05:57 on dist-test-slave-1gc3"
server_key: "a055a80d6d8d8e167582dfc861888275"
server_key_iv: "2ce34534e02e37c88b9863bc54f0c43f"
server_key_version: "encryptionkey@0"
I20260924 16:05:57.143555 7821 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 16:05:57.148425 7836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:57.150395 7821 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:05:57.150641 7821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "c2c01f5adef94944ab19f023ca2c858a"
format_stamp: "Formatted at 2026-09-24 16:05:57 on dist-test-slave-1gc3"
server_key: "a055a80d6d8d8e167582dfc861888275"
server_key_iv: "2ce34534e02e37c88b9863bc54f0c43f"
server_key_version: "encryptionkey@0"
I20260924 16:05:57.150909 7821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:57.168206 7821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:57.173702 7821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:57.174058 7821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:57.175341 7821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:57.177007 7821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:57.177095 7821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:57.177242 7821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:57.177305 7821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:57.215251 7821 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:32887
I20260924 16:05:57.215312 7951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:32887 every 8 connection(s)
I20260924 16:05:57.216857 7821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:05:57.224774 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 7821
I20260924 16:05:57.224970 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:05:57.225431 3931 external_mini_cluster.cc:1428] Setting key 8a7f822747a7a43c5fa8f5e24ba2a85f
Sep 24 16:05:57 dist-test-slave-1gc3 krb5kdc[7575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265957, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:57.246325 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.220121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:54781 (local address 127.3.214.254:38395)
0924 16:05:57.220769 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:57.220775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:57.222075 (+ 1300us) server_negotiation.cc:407] Connection header received
0924 16:05:57.224230 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:57.224236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:57.224346 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:57.224482 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:57.227156 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.228113 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.230366 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.230991 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.231149 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.236443 (+ 5294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:57.236452 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:57.236457 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:57.236516 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:57.241075 (+ 4559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:57.242508 (+ 1433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:57.242515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:57.242518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:57.242746 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:57.243521 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:57.243527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:57.243530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:57.244191 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:57.244409 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:57.245484 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0924 16:05:57.245822 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":321,"threads_started":1}
I20260924 16:05:57.247714 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.220868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38395 (local address 127.3.214.194:54781)
0924 16:05:57.221842 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:57.221909 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:57.223901 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:57.224930 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:57.224949 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:57.225832 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:57.226921 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.226954 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.228357 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.228363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.230068 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.230091 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.231296 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.231304 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.231666 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.232755 (+ 1089us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:57.232796 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:57.235976 (+ 3180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:57.241394 (+ 5418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:57.241408 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:57.241432 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:57.242322 (+ 890us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:57.242958 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:57.242965 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:57.242970 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:57.243296 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:57.244360 (+ 1064us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:57.244372 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:57.244962 (+ 590us) client_negotiation.cc:769] Sending connection context
0924 16:05:57.245349 (+ 387us) client_negotiation.cc:240] Negotiation successful
0924 16:05:57.246090 (+ 741us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":379,"threads_started":1}
WARNING: no policy specified for kudu/127.3.214.195@KRBTEST.COM; defaulting to no policy
I20260924 16:05:57.249899 7952 heartbeater.cc:347] Connected to a master server at 127.3.214.254:38395
I20260924 16:05:57.250384 7952 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:57.251284 7952 heartbeater.cc:511] Master 127.3.214.254:38395 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.195@KRBTEST.COM" created.
I20260924 16:05:57.253549 7623 ts_manager.cc:194] Registered new tserver with Master: c2c01f5adef94944ab19f023ca2c858a (127.3.214.194:32887)
I20260924 16:05:57.254877 7623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:54781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
I20260924 16:05:57.280143 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:38395
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:05:57.654850 7958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:05:57.655274 7958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:05:57.655440 7958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 16:05:57.655961 7815 heartbeater.cc:503] Master 127.3.214.254:38395 was elected leader, sending a full tablet report...
W20260924 16:05:57.663882 7958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:05:57.664050 7958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:05:57.664227 7958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:05:57.675779 7958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:38395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:05:57.677613 7958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:05:57.679446 7958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:05:57.695597 7963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:57.695814 7966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:05:57.696202 7964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:05:57.697333 7958 server_base.cc:1034] running on GCE node
I20260924 16:05:57.698288 7958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:05:57.699637 7958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:05:57.700901 7958 hybrid_clock.cc:648] HybridClock initialized: now 1790265957700827 us; error 62 us; skew 500 ppm
I20260924 16:05:57.701313 7958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:05:57 dist-test-slave-1gc3 krb5kdc[7575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265957, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:05:57.705788 7958 init.cc:377] Logged in from keytab as kudu/127.3.214.195@KRBTEST.COM (short username kudu)
I20260924 16:05:57.708074 7958 webserver.cc:460] Webserver started at http://127.3.214.195:45603/ using document root <none> and password file <none>
I20260924 16:05:57.709064 7958 fs_manager.cc:359] Metadata directory not provided
I20260924 16:05:57.709214 7958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:05:57.709589 7958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:05:57.713394 7958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "3ffc2655bd304c3caf7e8f27be48f069"
format_stamp: "Formatted at 2026-09-24 16:05:57 on dist-test-slave-1gc3"
server_key: "7bbd48e99ca8efbaa51dffab3db34785"
server_key_iv: "0af7c1623909692a0dd5401ea9d9b9d9"
server_key_version: "encryptionkey@0"
I20260924 16:05:57.714700 7958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "3ffc2655bd304c3caf7e8f27be48f069"
format_stamp: "Formatted at 2026-09-24 16:05:57 on dist-test-slave-1gc3"
server_key: "7bbd48e99ca8efbaa51dffab3db34785"
server_key_iv: "0af7c1623909692a0dd5401ea9d9b9d9"
server_key_version: "encryptionkey@0"
I20260924 16:05:57.721905 7958 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260924 16:05:57.726797 7973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:57.728713 7958 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:05:57.728935 7958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "3ffc2655bd304c3caf7e8f27be48f069"
format_stamp: "Formatted at 2026-09-24 16:05:57 on dist-test-slave-1gc3"
server_key: "7bbd48e99ca8efbaa51dffab3db34785"
server_key_iv: "0af7c1623909692a0dd5401ea9d9b9d9"
server_key_version: "encryptionkey@0"
I20260924 16:05:57.729182 7958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:05:57.748477 7958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:05:57.753832 7958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:05:57.754196 7958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:05:57.755431 7958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:05:57.757215 7958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:05:57.757308 7958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:57.757427 7958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:05:57.757481 7958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:05:57.796418 7958 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:41197
I20260924 16:05:57.796499 8087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:41197 every 8 connection(s)
I20260924 16:05:57.798038 7958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:05:57.806543 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 7958
I20260924 16:05:57.806790 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:05:57.807312 3931 external_mini_cluster.cc:1428] Setting key 519762c3b682c5908f37d58117996daf
Sep 24 16:05:57 dist-test-slave-1gc3 krb5kdc[7575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265957, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:57.823977 8090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.801455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:50057 (local address 127.3.214.254:38395)
0924 16:05:57.802206 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:57.802212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:57.803556 (+ 1344us) server_negotiation.cc:407] Connection header received
0924 16:05:57.805835 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:57.805846 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:57.805978 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:57.806085 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:57.808630 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.809669 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.811952 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.812400 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.812517 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.817009 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:57.817016 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:57.817020 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:57.817066 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:57.819921 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:57.820946 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:57.820951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:57.820953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:57.821111 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:57.821621 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:57.821627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:57.821630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:57.822057 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:57.822269 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:57.823303 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0924 16:05:57.823578 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":356,"threads_started":1}
I20260924 16:05:57.825352 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.802272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38395 (local address 127.3.214.195:50057)
0924 16:05:57.803311 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0924 16:05:57.803379 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 16:05:57.805440 (+ 2061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:05:57.806456 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:05:57.806467 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:05:57.807181 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:05:57.808366 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.808396 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.809914 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.809922 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.811772 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:05:57.811797 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.812675 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.812681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:05:57.812960 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.813839 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:05:57.813868 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:05:57.816637 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:05:57.820126 (+ 3489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:57.820137 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:57.820158 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:57.820769 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:57.821214 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:05:57.821218 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:05:57.821220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:05:57.821456 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:05:57.822273 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:05:57.822283 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:05:57.822805 (+ 522us) client_negotiation.cc:769] Sending connection context
0924 16:05:57.823157 (+ 352us) client_negotiation.cc:240] Negotiation successful
0924 16:05:57.823831 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":387,"threads_started":1}
I20260924 16:05:57.827175 8088 heartbeater.cc:347] Connected to a master server at 127.3.214.254:38395
I20260924 16:05:57.827533 8088 heartbeater.cc:464] Registering TS with master...
I20260924 16:05:57.828408 8088 heartbeater.cc:511] Master 127.3.214.254:38395 requested a full tablet report, sending...
I20260924 16:05:57.830361 7623 ts_manager.cc:194] Registered new tserver with Master: 3ffc2655bd304c3caf7e8f27be48f069 (127.3.214.195:41197)
I20260924 16:05:57.831271 7623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:50057
I20260924 16:05:57.838610 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 16:05:57 dist-test-slave-1gc3 krb5kdc[7575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 16:05:57 dist-test-slave-1gc3 krb5kdc[7575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.193@KRBTEST.COM
I20260924 16:05:57.903995 8097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.883143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40106 (local address 127.3.214.193:46043)
0924 16:05:57.884644 (+ 1501us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:57.884654 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:57.884686 (+ 32us) server_negotiation.cc:407] Connection header received
0924 16:05:57.884758 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:57.884766 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:57.885008 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:57.885164 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:57.887187 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.888332 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.890315 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.891306 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.891509 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.896381 (+ 4872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:57.896406 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:57.896432 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:57.896487 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:57.900269 (+ 3782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:57.901405 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:57.901415 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:57.901417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:57.901599 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:57.902197 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:57.902201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:57.902203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:57.902631 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:57.902908 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:57.903413 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 16:05:57.903661 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1330,"thread_start_us":1150,"threads_started":1}
Sep 24 16:05:57 dist-test-slave-1gc3 krb5kdc[7575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:05:57.926313 8090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.910292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47108 (local address 127.3.214.254:38395)
0924 16:05:57.910551 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:57.910557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:57.910797 (+ 240us) server_negotiation.cc:407] Connection header received
0924 16:05:57.910996 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:57.911002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:57.911078 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:57.911164 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:57.912593 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.913354 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.914702 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.915132 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.915243 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.920223 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:57.920229 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:57.920233 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:57.920281 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:57.923403 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:57.924233 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:57.924237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:57.924240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:57.924383 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:57.924769 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:57.924773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:57.924775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:57.925143 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:57.925399 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:57.925714 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 16:05:57.925953 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260924 16:05:57.927211 7623 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47108
I20260924 16:05:57.928525 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 7684
I20260924 16:05:57.944327 8097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:57.929833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40122 (local address 127.3.214.193:46043)
0924 16:05:57.930182 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:57.930192 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:57.930491 (+ 299us) server_negotiation.cc:407] Connection header received
0924 16:05:57.930717 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:57.930726 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:57.930875 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:57.930978 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:57.932471 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.933277 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.934753 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:57.935223 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:57.935351 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:57.938025 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:57.938031 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:57.938035 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:57.938079 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:57.941215 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:57.942173 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:57.942179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:57.942181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:57.942352 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:57.942725 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:57.942731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:57.942735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:57.943052 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:57.943376 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:57.943674 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 16:05:57.943949 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260924 16:05:57.945180 7810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:58.038985 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 7684
I20260924 16:05:58.058804 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 7821
Sep 24 16:05:58 dist-test-slave-1gc3 krb5kdc[7575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.194@KRBTEST.COM
I20260924 16:05:58.077690 8102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:58.059931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58900 (local address 127.3.214.194:32887)
0924 16:05:58.060481 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:58.060490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:58.060522 (+ 32us) server_negotiation.cc:407] Connection header received
0924 16:05:58.060597 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:58.060605 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:58.060837 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:58.060987 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:58.062376 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:58.063444 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:58.064740 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:58.065323 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:58.065432 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:58.070237 (+ 4805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:58.070256 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:58.070276 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:58.070324 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:58.073824 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:58.075013 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:58.075029 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:58.075032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:58.075282 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:58.075725 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:58.075731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:58.075735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:58.076218 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:58.076606 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:58.076922 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 16:05:58.077262 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":262,"threads_started":1}
I20260924 16:05:58.079224 7947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:58.172096 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 7821
I20260924 16:05:58.189208 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 7958
Sep 24 16:05:58 dist-test-slave-1gc3 krb5kdc[7575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.195@KRBTEST.COM
I20260924 16:05:58.210246 8104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:05:58.190763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37768 (local address 127.3.214.195:41197)
0924 16:05:58.191665 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0924 16:05:58.191673 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:05:58.191710 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:05:58.191793 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:05:58.191800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:05:58.192032 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:05:58.192195 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:05:58.193981 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:58.195283 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:58.196870 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:05:58.197490 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:05:58.197613 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:05:58.203040 (+ 5427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:05:58.203054 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:05:58.203075 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:05:58.203121 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:05:58.206424 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:58.207387 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:58.207396 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:58.207399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:58.207617 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:05:58.208135 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:05:58.208139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:05:58.208141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:05:58.208733 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:05:58.209025 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 16:05:58.209402 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 16:05:58.209705 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":477,"threads_started":1}
I20260924 16:05:58.211995 8083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:58.306473 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 7958
I20260924 16:05:58.325766 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 7588
I20260924 16:05:58.327437 7653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:05:58.440091 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 7588
2026-09-24T16:05:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4351 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:05:58 dist-test-slave-1gc3 krb5kdc[8109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:05:58 dist-test-slave-1gc3 krb5kdc[8109](info): set up 2 sockets
Sep 24 16:05:58 dist-test-slave-1gc3 krb5kdc[8109](info): commencing operation
krb5kdc: starting...
W20260924 16:05:59.493191 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:05:59 dist-test-slave-1gc3 krb5kdc[8109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:05:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:05:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:05:59.729734 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:46791
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:38055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:46791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:00.081840 8122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:00.082196 8122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:00.082340 8122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:00.089816 8122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:00.089955 8122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:00.090024 8122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:00.090124 8122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:00.090194 8122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:00.100751 8122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:38055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:46791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:46791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:00.102264 8122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:00.103670 8122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:00.116351 8127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:00.116528 8128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:00.116544 8130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:00.117777 8122 server_base.cc:1034] running on GCE node
I20260924 16:06:00.118741 8122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:00.120353 8122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:00.121652 8122 hybrid_clock.cc:648] HybridClock initialized: now 1790265960121595 us; error 50 us; skew 500 ppm
I20260924 16:06:00.122072 8122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:00 dist-test-slave-1gc3 krb5kdc[8109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265960, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:00.125552 8122 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:06:00.127000 8122 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:00.127117 8122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:00.127468 8122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:00.130815 8122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "a0366fc83bbe490e8271c2bcfeab241f"
format_stamp: "Formatted at 2026-09-24 16:06:00 on dist-test-slave-1gc3"
server_key: "eda8316167d7adbae46dd875b622922f"
server_key_iv: "d0ebf398f6c3d4255adddb28635aec5f"
server_key_version: "encryptionkey@0"
I20260924 16:06:00.131742 8122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "a0366fc83bbe490e8271c2bcfeab241f"
format_stamp: "Formatted at 2026-09-24 16:06:00 on dist-test-slave-1gc3"
server_key: "eda8316167d7adbae46dd875b622922f"
server_key_iv: "d0ebf398f6c3d4255adddb28635aec5f"
server_key_version: "encryptionkey@0"
I20260924 16:06:00.138195 8122 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 16:06:00.142759 8136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:00.144690 8122 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:06:00.144940 8122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "a0366fc83bbe490e8271c2bcfeab241f"
format_stamp: "Formatted at 2026-09-24 16:06:00 on dist-test-slave-1gc3"
server_key: "eda8316167d7adbae46dd875b622922f"
server_key_iv: "d0ebf398f6c3d4255adddb28635aec5f"
server_key_version: "encryptionkey@0"
I20260924 16:06:00.145196 8122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:00.166951 8122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:00.171540 8122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:00.171823 8122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:00.192667 8122 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:46791
I20260924 16:06:00.192653 8188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:46791 every 8 connection(s)
I20260924 16:06:00.193722 8122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:00.194386 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 8122
I20260924 16:06:00.194679 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:06:00.195538 3931 external_mini_cluster.cc:1428] Setting key c7821b4b4dfd8790ce47f25f9c08b805
I20260924 16:06:00.198508 8189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:00.213312 8189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f: Bootstrap starting.
Sep 24 16:06:00 dist-test-slave-1gc3 krb5kdc[8109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:06:00.217582 8189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:00.219062 8189 log.cc:824] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:00.223029 8189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f: No bootstrap required, opened a new log
I20260924 16:06:00.224166 8192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:00.198414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44600 (local address 127.3.214.254:46791)
0924 16:06:00.199674 (+ 1260us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:00.199694 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:00.199767 (+ 73us) server_negotiation.cc:407] Connection header received
0924 16:06:00.202134 (+ 2367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:00.202171 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:00.202830 (+ 659us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:00.203376 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:00.205193 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.207383 (+ 2190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.208869 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.209998 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.210300 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:00.214712 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:00.214726 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:00.214749 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:00.214817 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:00.218946 (+ 4129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:00.219955 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:00.219962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:00.219969 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:00.220174 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:00.220662 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:00.220667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:00.220669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:00.221467 (+ 798us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:00.221771 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:00.222315 (+ 544us) server_negotiation.cc:299] Negotiation successful
0924 16:06:00.222789 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":467,"threads_started":1}
I20260924 16:06:00.231760 8189 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0366fc83bbe490e8271c2bcfeab241f" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 46791 } }
I20260924 16:06:00.232175 8189 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:00.232254 8189 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0366fc83bbe490e8271c2bcfeab241f, State: Initialized, Role: FOLLOWER
I20260924 16:06:00.232872 8189 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0366fc83bbe490e8271c2bcfeab241f" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 46791 } }
I20260924 16:06:00.233108 8189 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:00.233212 8189 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:00.233351 8189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:00.235994 8189 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0366fc83bbe490e8271c2bcfeab241f" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 46791 } }
I20260924 16:06:00.236567 8189 leader_election.cc:302] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0366fc83bbe490e8271c2bcfeab241f; no voters:
I20260924 16:06:00.237231 8189 leader_election.cc:288] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:00.237541 8194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:00.238772 8194 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f [term 1 LEADER]: Becoming Leader. State: Replica: a0366fc83bbe490e8271c2bcfeab241f, State: Running, Role: LEADER
I20260924 16:06:00.239521 8194 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0366fc83bbe490e8271c2bcfeab241f" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 46791 } }
I20260924 16:06:00.239812 8189 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:00.243505 8195 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0366fc83bbe490e8271c2bcfeab241f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0366fc83bbe490e8271c2bcfeab241f" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 46791 } } }
I20260924 16:06:00.243783 8196 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0366fc83bbe490e8271c2bcfeab241f. Latest consensus state: current_term: 1 leader_uuid: "a0366fc83bbe490e8271c2bcfeab241f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0366fc83bbe490e8271c2bcfeab241f" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 46791 } } }
I20260924 16:06:00.244182 8195 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:00.244201 8196 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:00.249224 8202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:00.256376 8202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:00.267983 8202 catalog_manager.cc:1347] Generated new cluster ID: 9e75257f23174256991e4eb7a0818fab
I20260924 16:06:00.268143 8202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:00.317085 8202 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:00.319635 8202 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:00.334733 8202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f: Generated new TSK 0
I20260924 16:06:00.335789 8202 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.214.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
I20260924 16:06:00.403342 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:46791
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:38055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:00.794298 8215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:00.794701 8215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:00.794878 8215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:00.803216 8215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:00.803388 8215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:00.803567 8215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:06:00.815209 8215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:38055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:46791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:00.817010 8215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:00.818689 8215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:00.835242 8221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:00.835388 8223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:00.835613 8220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:00.836704 8215 server_base.cc:1034] running on GCE node
I20260924 16:06:00.837577 8215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:00.838959 8215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:00.840271 8215 hybrid_clock.cc:648] HybridClock initialized: now 1790265960840207 us; error 45 us; skew 500 ppm
I20260924 16:06:00.840749 8215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:00 dist-test-slave-1gc3 krb5kdc[8109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265960, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:00.850754 8215 init.cc:377] Logged in from keytab as kudu/127.3.214.193@KRBTEST.COM (short username kudu)
I20260924 16:06:00.852349 8215 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:00.852512 8215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:00.852870 8215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:00.856333 8215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "8dfc859e16d248a6b2f6ac8e3fc49bcc"
format_stamp: "Formatted at 2026-09-24 16:06:00 on dist-test-slave-1gc3"
server_key: "ab78a4014c9a51737fd61994ccf1db41"
server_key_iv: "282fe4329ef40b039defe63b27ff28a7"
server_key_version: "encryptionkey@0"
I20260924 16:06:00.857488 8215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "8dfc859e16d248a6b2f6ac8e3fc49bcc"
format_stamp: "Formatted at 2026-09-24 16:06:00 on dist-test-slave-1gc3"
server_key: "ab78a4014c9a51737fd61994ccf1db41"
server_key_iv: "282fe4329ef40b039defe63b27ff28a7"
server_key_version: "encryptionkey@0"
I20260924 16:06:00.863833 8215 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260924 16:06:00.868291 8229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:00.870064 8215 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:06:00.870364 8215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "8dfc859e16d248a6b2f6ac8e3fc49bcc"
format_stamp: "Formatted at 2026-09-24 16:06:00 on dist-test-slave-1gc3"
server_key: "ab78a4014c9a51737fd61994ccf1db41"
server_key_iv: "282fe4329ef40b039defe63b27ff28a7"
server_key_version: "encryptionkey@0"
I20260924 16:06:00.870635 8215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:00.908217 8215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:00.913154 8215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:00.913478 8215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:00.914636 8215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:00.916489 8215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:00.916618 8215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:00.916750 8215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:00.916857 8215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:00.955971 8215 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:44231
I20260924 16:06:00.956060 8342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:44231 every 8 connection(s)
I20260924 16:06:00.956925 8215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:06:00.958271 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 8215
I20260924 16:06:00.958544 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:06:00.959412 3931 external_mini_cluster.cc:1428] Setting key 81528e2b66b07b5955fc33bee6dbf16b
Sep 24 16:06:00 dist-test-slave-1gc3 krb5kdc[8109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265960, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:06:00.981498 8345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:00.960530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:57587 (local address 127.3.214.254:46791)
0924 16:06:00.961124 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:00.961132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:00.962359 (+ 1227us) server_negotiation.cc:407] Connection header received
0924 16:06:00.964404 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:00.964410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:00.964508 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:00.964645 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:00.966390 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.967443 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.969283 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.969765 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.969895 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:00.974137 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:00.974146 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:00.974151 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:00.974193 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:00.976989 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:00.978031 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:00.978043 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:00.978046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:00.978226 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:00.978810 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:00.978817 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:00.978821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:00.979289 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:00.979749 (+ 460us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:00.980809 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0924 16:06:00.981118 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":288,"threads_started":1}
I20260924 16:06:00.982607 8346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:00.961223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:46791 (local address 127.3.214.193:57587)
0924 16:06:00.962155 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:00.962206 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:00.964089 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:00.964908 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:00.964916 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:00.965505 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:00.966177 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:00.966212 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.967645 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.967649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:00.969021 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:00.969039 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:00.970037 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:00.970044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:00.970402 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:00.971190 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:00.971217 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:00.973790 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:00.977281 (+ 3491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:00.977293 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:00.977320 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:00.977869 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:00.978343 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:00.978348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:00.978353 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:00.978597 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:00.979722 (+ 1125us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:00.979732 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:00.980266 (+ 534us) client_negotiation.cc:769] Sending connection context
0924 16:06:00.980644 (+ 378us) client_negotiation.cc:240] Negotiation successful
0924 16:06:00.981260 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":370,"threads_started":1}
WARNING: no policy specified for kudu/127.3.214.194@KRBTEST.COM; defaulting to no policy
I20260924 16:06:00.984699 8343 heartbeater.cc:347] Connected to a master server at 127.3.214.254:46791
I20260924 16:06:00.985152 8343 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.194@KRBTEST.COM" created.
I20260924 16:06:00.986157 8343 heartbeater.cc:511] Master 127.3.214.254:46791 requested a full tablet report, sending...
I20260924 16:06:00.989465 8153 ts_manager.cc:194] Registered new tserver with Master: 8dfc859e16d248a6b2f6ac8e3fc49bcc (127.3.214.193:44231)
I20260924 16:06:00.991773 8153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:57587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
I20260924 16:06:01.015969 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:46791
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:38055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:01.373553 8350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:01.373878 8350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:01.374056 8350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:01.381693 8350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:01.381836 8350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:01.381994 8350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:06:01.392354 8350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:38055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:46791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:01.393795 8350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:01.395229 8350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:01.409967 8356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:01.409912 8358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:01.409868 8355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:01.411033 8350 server_base.cc:1034] running on GCE node
I20260924 16:06:01.411721 8350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:01.412822 8350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:01.414080 8350 hybrid_clock.cc:648] HybridClock initialized: now 1790265961413991 us; error 77 us; skew 500 ppm
I20260924 16:06:01.414515 8350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:01 dist-test-slave-1gc3 krb5kdc[8109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265961, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:01.417961 8350 init.cc:377] Logged in from keytab as kudu/127.3.214.194@KRBTEST.COM (short username kudu)
I20260924 16:06:01.419358 8350 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:01.419464 8350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:01.419775 8350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:01.423032 8350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "4d40c54873384912a9dc09622d3aad3c"
format_stamp: "Formatted at 2026-09-24 16:06:01 on dist-test-slave-1gc3"
server_key: "03d9dd559b9cef57e5af31e2439baa0b"
server_key_iv: "b3555ceac092e032231301d0da47f189"
server_key_version: "encryptionkey@0"
I20260924 16:06:01.423913 8350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "4d40c54873384912a9dc09622d3aad3c"
format_stamp: "Formatted at 2026-09-24 16:06:01 on dist-test-slave-1gc3"
server_key: "03d9dd559b9cef57e5af31e2439baa0b"
server_key_iv: "b3555ceac092e032231301d0da47f189"
server_key_version: "encryptionkey@0"
I20260924 16:06:01.429891 8350 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 16:06:01.434471 8364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:01.436298 8350 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:06:01.436568 8350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "4d40c54873384912a9dc09622d3aad3c"
format_stamp: "Formatted at 2026-09-24 16:06:01 on dist-test-slave-1gc3"
server_key: "03d9dd559b9cef57e5af31e2439baa0b"
server_key_iv: "b3555ceac092e032231301d0da47f189"
server_key_version: "encryptionkey@0"
I20260924 16:06:01.436817 8350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:01.458305 8350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:01.463238 8350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:01.463542 8350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:01.464633 8350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:01.466177 8350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:01.466289 8350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:01.466429 8350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:01.466495 8350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:01.502519 8350 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:44751
I20260924 16:06:01.502655 8477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:44751 every 8 connection(s)
I20260924 16:06:01.503410 8350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:06:01.512141 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 8350
I20260924 16:06:01.512392 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:06:01.513028 3931 external_mini_cluster.cc:1428] Setting key 29f3f77fb1b6c57dcf851bc869b18021
Sep 24 16:06:01 dist-test-slave-1gc3 krb5kdc[8109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265961, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:06:01.529016 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:01.506518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:47689 (local address 127.3.214.254:46791)
0924 16:06:01.507125 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:01.507132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:01.508180 (+ 1048us) server_negotiation.cc:407] Connection header received
0924 16:06:01.510475 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:01.510480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:01.510574 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:01.510705 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:01.513140 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.514153 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.516041 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.516526 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.516650 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:01.521311 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:01.521317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:01.521321 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:01.521366 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:01.524835 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:01.525998 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:01.526004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:01.526008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:01.526210 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:01.526835 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:01.526840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:01.526844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:01.527243 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:01.527405 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:01.528390 (+ 985us) server_negotiation.cc:299] Negotiation successful
0924 16:06:01.528658 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":277,"threads_started":1}
I20260924 16:06:01.530361 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:01.507053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:46791 (local address 127.3.214.194:47689)
0924 16:06:01.507965 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:01.508016 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:01.510069 (+ 2053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:01.511026 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:01.511037 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:01.511797 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:01.512865 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:01.512896 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.514386 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.514391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:01.515836 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:01.515867 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:01.516803 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:01.516812 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:01.517076 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:01.517920 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:01.517948 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:01.520922 (+ 2974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:01.525089 (+ 4167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:01.525100 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:01.525126 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:01.525831 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:01.526387 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:01.526391 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:01.526394 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:01.526705 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:01.527427 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:01.527435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:01.527932 (+ 497us) client_negotiation.cc:769] Sending connection context
0924 16:06:01.528231 (+ 299us) client_negotiation.cc:240] Negotiation successful
0924 16:06:01.528853 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":320,"threads_started":1}
I20260924 16:06:01.532418 8478 heartbeater.cc:347] Connected to a master server at 127.3.214.254:46791
I20260924 16:06:01.532823 8478 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:01.533651 8478 heartbeater.cc:511] Master 127.3.214.254:46791 requested a full tablet report, sending...
I20260924 16:06:01.535658 8153 ts_manager.cc:194] Registered new tserver with Master: 4d40c54873384912a9dc09622d3aad3c (127.3.214.194:44751)
WARNING: no policy specified for kudu/127.3.214.195@KRBTEST.COM; defaulting to no policy
I20260924 16:06:01.536645 8153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:47689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
I20260924 16:06:01.565599 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:46791
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:38055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:01.966519 8484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:01.966945 8484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:01.967089 8484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:01.975597 8484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:01.975783 8484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:01.975996 8484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:06:01.987881 8484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:38055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:46791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:01.989738 8484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:01.991436 8484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:06:01.996701 8343 heartbeater.cc:503] Master 127.3.214.254:46791 was elected leader, sending a full tablet report...
W20260924 16:06:02.009100 8492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:02.009058 8490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:02.009127 8489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:02.010339 8484 server_base.cc:1034] running on GCE node
I20260924 16:06:02.011201 8484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:02.012410 8484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:02.013829 8484 hybrid_clock.cc:648] HybridClock initialized: now 1790265962013639 us; error 73 us; skew 500 ppm
I20260924 16:06:02.014329 8484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:02 dist-test-slave-1gc3 krb5kdc[8109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265962, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:02.018198 8484 init.cc:377] Logged in from keytab as kudu/127.3.214.195@KRBTEST.COM (short username kudu)
I20260924 16:06:02.019749 8484 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:02.019879 8484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:02.020237 8484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:02.023787 8484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "244e4d158ca14d05ae58ad76330253ad"
format_stamp: "Formatted at 2026-09-24 16:06:02 on dist-test-slave-1gc3"
server_key: "06663736d181cb751ed34d0a9da0be5b"
server_key_iv: "a1fcbd6c5cb91a9b263ed15e4c1fe97f"
server_key_version: "encryptionkey@0"
I20260924 16:06:02.024780 8484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "244e4d158ca14d05ae58ad76330253ad"
format_stamp: "Formatted at 2026-09-24 16:06:02 on dist-test-slave-1gc3"
server_key: "06663736d181cb751ed34d0a9da0be5b"
server_key_iv: "a1fcbd6c5cb91a9b263ed15e4c1fe97f"
server_key_version: "encryptionkey@0"
I20260924 16:06:02.032014 8484 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260924 16:06:02.036775 8498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:02.038934 8484 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:06:02.039193 8484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "244e4d158ca14d05ae58ad76330253ad"
format_stamp: "Formatted at 2026-09-24 16:06:02 on dist-test-slave-1gc3"
server_key: "06663736d181cb751ed34d0a9da0be5b"
server_key_iv: "a1fcbd6c5cb91a9b263ed15e4c1fe97f"
server_key_version: "encryptionkey@0"
I20260924 16:06:02.039618 8484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:02.063364 8484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:02.068391 8484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:02.068673 8484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:02.069842 8484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:02.071496 8484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:02.071596 8484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:02.071707 8484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:02.071776 8484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:02.108490 8484 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:45493
I20260924 16:06:02.108583 8613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:45493 every 8 connection(s)
I20260924 16:06:02.109443 8484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:06:02.112236 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 8484
I20260924 16:06:02.112474 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:06:02.112999 3931 external_mini_cluster.cc:1428] Setting key 2c4c1d1cfbabe15f34f96720b78a9471
Sep 24 16:06:02 dist-test-slave-1gc3 krb5kdc[8109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265962, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:06:02.132138 8616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.112720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:47143 (local address 127.3.214.254:46791)
0924 16:06:02.113395 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:02.113404 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:02.114376 (+ 972us) server_negotiation.cc:407] Connection header received
0924 16:06:02.116568 (+ 2192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:02.116572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:02.116657 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:02.116775 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:02.118657 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.119532 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.121137 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.121596 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.121701 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:02.125475 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:02.125480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:02.125485 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:02.125525 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:02.128880 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:02.129830 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:02.129834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:02.129837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:02.129977 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:02.130396 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:02.130399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:02.130401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:02.130724 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:02.130855 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:02.131588 (+ 733us) server_negotiation.cc:299] Negotiation successful
0924 16:06:02.131814 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":311,"threads_started":1}
I20260924 16:06:02.133133 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.113279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:46791 (local address 127.3.214.195:47143)
0924 16:06:02.114172 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:02.114233 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:02.116253 (+ 2020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:02.117086 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:02.117096 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:02.117693 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:02.118402 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:02.118472 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.119677 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.119681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:02.120992 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:02.121007 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.121811 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.121816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:02.122029 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:02.122769 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:02.122794 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:02.125208 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:02.129064 (+ 3856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:02.129074 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:02.129092 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:02.129698 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:02.130069 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:02.130072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:02.130074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:02.130291 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:02.130844 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:02.130851 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:02.131220 (+ 369us) client_negotiation.cc:769] Sending connection context
0924 16:06:02.131461 (+ 241us) client_negotiation.cc:240] Negotiation successful
0924 16:06:02.131913 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":347,"threads_started":1}
I20260924 16:06:02.134951 8614 heartbeater.cc:347] Connected to a master server at 127.3.214.254:46791
I20260924 16:06:02.135327 8614 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:02.136174 8614 heartbeater.cc:511] Master 127.3.214.254:46791 requested a full tablet report, sending...
I20260924 16:06:02.138031 8153 ts_manager.cc:194] Registered new tserver with Master: 244e4d158ca14d05ae58ad76330253ad (127.3.214.195:45493)
I20260924 16:06:02.139025 8153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:47143
I20260924 16:06:02.142859 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:06:02.162935 8616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.148541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44608 (local address 127.3.214.254:46791)
0924 16:06:02.148837 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:02.148845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:02.149329 (+ 484us) server_negotiation.cc:407] Connection header received
0924 16:06:02.149530 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:02.149535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:02.149650 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:02.149762 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:02.151438 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.152303 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.153587 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.154060 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.154249 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:02.156638 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:02.156643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:02.156648 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:02.156705 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:02.159824 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:02.160646 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:02.160651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:02.160654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:02.160837 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:02.161270 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:02.161273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:02.161275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:02.161707 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:02.161928 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:02.162315 (+ 387us) server_negotiation.cc:299] Negotiation successful
0924 16:06:02.162555 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260924 16:06:02.168272 8153 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:06:02.176707 8153 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:06:02.219165 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.194821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40406 (local address 127.3.214.194:44751)
0924 16:06:02.206809 (+ 11988us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:02.206820 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:02.208073 (+ 1253us) server_negotiation.cc:407] Connection header received
0924 16:06:02.208401 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:02.208411 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:02.208701 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:02.208888 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:02.209965 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.211817 (+ 1852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.215631 (+ 3814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.217244 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.217443 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:02.217672 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:02.218617 (+ 945us) server_negotiation.cc:299] Negotiation successful
0924 16:06:02.218831 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11772,"thread_start_us":11659,"threads_started":1}
I20260924 16:06:02.220461 8627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.194511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:44751 (local address 127.0.0.1:40406)
0924 16:06:02.207941 (+ 13430us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:02.207962 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:02.208193 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:02.209221 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:02.209229 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:02.209263 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:02.209738 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:02.209756 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.212078 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.212084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:02.215320 (+ 3236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:02.215352 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.217887 (+ 2535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.217897 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:02.218291 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:02.218355 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 16:06:02.219719 (+ 1364us) client_negotiation.cc:240] Negotiation successful
0924 16:06:02.219982 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13280,"mutex_wait_us":63,"thread_start_us":12316,"threads_started":1}
I20260924 16:06:02.225651 8413 tablet_service.cc:1467] Processing CreateTablet for tablet 1cb4a81341434e52ab65dd5de3ab90ba (DEFAULT_TABLE table=test-table [id=05b2bb24a4674e508184d6886e4c56f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:02.227378 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cb4a81341434e52ab65dd5de3ab90ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:02.229030 8628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.194340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32832 (local address 127.3.214.193:44231)
0924 16:06:02.216187 (+ 21847us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:02.216201 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:02.216248 (+ 47us) server_negotiation.cc:407] Connection header received
0924 16:06:02.216408 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:02.216548 (+ 140us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:02.216978 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:02.218604 (+ 1626us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:02.218964 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.223015 (+ 4051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.225750 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.227544 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.227781 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:02.228115 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:02.228402 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 16:06:02.228619 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21582,"thread_start_us":22205,"threads_started":1}
I20260924 16:06:02.229115 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.193962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:44231 (local address 127.0.0.1:32832)
0924 16:06:02.205745 (+ 11783us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:02.205814 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:02.206067 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:02.217346 (+ 11279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:02.217354 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:02.217396 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:02.218306 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:02.218328 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.223201 (+ 4873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.223208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:02.225491 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:02.225509 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.227895 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.227903 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:02.228162 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:02.228211 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 16:06:02.228321 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 16:06:02.228536 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11432,"mutex_wait_us":26,"thread_start_us":13699,"threads_started":1}
I20260924 16:06:02.233433 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.209796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:45493 (local address 127.0.0.1:60506)
0924 16:06:02.214512 (+ 4716us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:02.214549 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:02.214790 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:02.223649 (+ 8859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:02.223655 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:02.223695 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:02.224229 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:02.224245 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.227244 (+ 2999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.227252 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:02.229768 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:02.229800 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.232394 (+ 2594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.232401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:02.232574 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:02.232615 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 16:06:02.232747 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 16:06:02.232940 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4464,"mutex_wait_us":4,"thread_start_us":5127,"threads_started":1}
I20260924 16:06:02.233565 8631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.209740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60506 (local address 127.3.214.195:45493)
0924 16:06:02.222867 (+ 13127us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:02.222878 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:02.222927 (+ 49us) server_negotiation.cc:407] Connection header received
0924 16:06:02.223054 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:02.223063 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:02.223369 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:02.223642 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:02.224608 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.227016 (+ 2408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.230475 (+ 3459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.232106 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.232289 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:02.232570 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:02.232897 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 16:06:02.233148 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12789,"thread_start_us":12586,"threads_started":1}
I20260924 16:06:02.234995 8278 tablet_service.cc:1467] Processing CreateTablet for tablet 1cb4a81341434e52ab65dd5de3ab90ba (DEFAULT_TABLE table=test-table [id=05b2bb24a4674e508184d6886e4c56f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:02.236977 8278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cb4a81341434e52ab65dd5de3ab90ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:02.238637 8548 tablet_service.cc:1467] Processing CreateTablet for tablet 1cb4a81341434e52ab65dd5de3ab90ba (DEFAULT_TABLE table=test-table [id=05b2bb24a4674e508184d6886e4c56f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:02.240509 8548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cb4a81341434e52ab65dd5de3ab90ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:02.253187 8633 tablet_bootstrap.cc:492] T 1cb4a81341434e52ab65dd5de3ab90ba P 4d40c54873384912a9dc09622d3aad3c: Bootstrap starting.
I20260924 16:06:02.257815 8633 tablet_bootstrap.cc:654] T 1cb4a81341434e52ab65dd5de3ab90ba P 4d40c54873384912a9dc09622d3aad3c: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:02.259586 8633 log.cc:824] T 1cb4a81341434e52ab65dd5de3ab90ba P 4d40c54873384912a9dc09622d3aad3c: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:02.259559 8635 tablet_bootstrap.cc:492] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc: Bootstrap starting.
I20260924 16:06:02.261716 8634 tablet_bootstrap.cc:492] T 1cb4a81341434e52ab65dd5de3ab90ba P 244e4d158ca14d05ae58ad76330253ad: Bootstrap starting.
I20260924 16:06:02.263665 8633 tablet_bootstrap.cc:492] T 1cb4a81341434e52ab65dd5de3ab90ba P 4d40c54873384912a9dc09622d3aad3c: No bootstrap required, opened a new log
I20260924 16:06:02.264009 8633 ts_tablet_manager.cc:1397] T 1cb4a81341434e52ab65dd5de3ab90ba P 4d40c54873384912a9dc09622d3aad3c: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260924 16:06:02.264209 8635 tablet_bootstrap.cc:654] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:02.265482 8635 log.cc:824] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:02.266135 8634 tablet_bootstrap.cc:654] T 1cb4a81341434e52ab65dd5de3ab90ba P 244e4d158ca14d05ae58ad76330253ad: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:02.267691 8634 log.cc:824] T 1cb4a81341434e52ab65dd5de3ab90ba P 244e4d158ca14d05ae58ad76330253ad: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:02.269358 8635 tablet_bootstrap.cc:492] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc: No bootstrap required, opened a new log
I20260924 16:06:02.269726 8635 ts_tablet_manager.cc:1397] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260924 16:06:02.275250 8633 raft_consensus.cc:348] T 1cb4a81341434e52ab65dd5de3ab90ba P 4d40c54873384912a9dc09622d3aad3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc859e16d248a6b2f6ac8e3fc49bcc" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 44231 } } peers { permanent_uuid: "244e4d158ca14d05ae58ad76330253ad" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 45493 } } peers { permanent_uuid: "4d40c54873384912a9dc09622d3aad3c" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 44751 } }
I20260924 16:06:02.275790 8633 raft_consensus.cc:374] T 1cb4a81341434e52ab65dd5de3ab90ba P 4d40c54873384912a9dc09622d3aad3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:02.275954 8633 raft_consensus.cc:729] T 1cb4a81341434e52ab65dd5de3ab90ba P 4d40c54873384912a9dc09622d3aad3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d40c54873384912a9dc09622d3aad3c, State: Initialized, Role: FOLLOWER
I20260924 16:06:02.276722 8633 consensus_queue.cc:260] T 1cb4a81341434e52ab65dd5de3ab90ba P 4d40c54873384912a9dc09622d3aad3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc859e16d248a6b2f6ac8e3fc49bcc" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 44231 } } peers { permanent_uuid: "244e4d158ca14d05ae58ad76330253ad" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 45493 } } peers { permanent_uuid: "4d40c54873384912a9dc09622d3aad3c" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 44751 } }
I20260924 16:06:02.278139 8634 tablet_bootstrap.cc:492] T 1cb4a81341434e52ab65dd5de3ab90ba P 244e4d158ca14d05ae58ad76330253ad: No bootstrap required, opened a new log
I20260924 16:06:02.278589 8634 ts_tablet_manager.cc:1397] T 1cb4a81341434e52ab65dd5de3ab90ba P 244e4d158ca14d05ae58ad76330253ad: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.007s
I20260924 16:06:02.279127 8478 heartbeater.cc:503] Master 127.3.214.254:46791 was elected leader, sending a full tablet report...
I20260924 16:06:02.279798 8633 ts_tablet_manager.cc:1428] T 1cb4a81341434e52ab65dd5de3ab90ba P 4d40c54873384912a9dc09622d3aad3c: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
I20260924 16:06:02.278640 8635 raft_consensus.cc:348] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc859e16d248a6b2f6ac8e3fc49bcc" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 44231 } } peers { permanent_uuid: "244e4d158ca14d05ae58ad76330253ad" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 45493 } } peers { permanent_uuid: "4d40c54873384912a9dc09622d3aad3c" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 44751 } }
I20260924 16:06:02.280444 8635 raft_consensus.cc:374] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:02.280586 8635 raft_consensus.cc:729] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dfc859e16d248a6b2f6ac8e3fc49bcc, State: Initialized, Role: FOLLOWER
I20260924 16:06:02.281926 8635 consensus_queue.cc:260] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc859e16d248a6b2f6ac8e3fc49bcc" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 44231 } } peers { permanent_uuid: "244e4d158ca14d05ae58ad76330253ad" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 45493 } } peers { permanent_uuid: "4d40c54873384912a9dc09622d3aad3c" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 44751 } }
I20260924 16:06:02.289506 8634 raft_consensus.cc:348] T 1cb4a81341434e52ab65dd5de3ab90ba P 244e4d158ca14d05ae58ad76330253ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc859e16d248a6b2f6ac8e3fc49bcc" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 44231 } } peers { permanent_uuid: "244e4d158ca14d05ae58ad76330253ad" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 45493 } } peers { permanent_uuid: "4d40c54873384912a9dc09622d3aad3c" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 44751 } }
I20260924 16:06:02.289923 8634 raft_consensus.cc:374] T 1cb4a81341434e52ab65dd5de3ab90ba P 244e4d158ca14d05ae58ad76330253ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:02.290149 8634 raft_consensus.cc:729] T 1cb4a81341434e52ab65dd5de3ab90ba P 244e4d158ca14d05ae58ad76330253ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244e4d158ca14d05ae58ad76330253ad, State: Initialized, Role: FOLLOWER
I20260924 16:06:02.290891 8634 consensus_queue.cc:260] T 1cb4a81341434e52ab65dd5de3ab90ba P 244e4d158ca14d05ae58ad76330253ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc859e16d248a6b2f6ac8e3fc49bcc" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 44231 } } peers { permanent_uuid: "244e4d158ca14d05ae58ad76330253ad" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 45493 } } peers { permanent_uuid: "4d40c54873384912a9dc09622d3aad3c" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 44751 } }
I20260924 16:06:02.291863 8635 ts_tablet_manager.cc:1428] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc: Time spent starting tablet: real 0.022s user 0.010s sys 0.010s
I20260924 16:06:02.292886 8614 heartbeater.cc:503] Master 127.3.214.254:46791 was elected leader, sending a full tablet report...
I20260924 16:06:02.293668 8634 ts_tablet_manager.cc:1428] T 1cb4a81341434e52ab65dd5de3ab90ba P 244e4d158ca14d05ae58ad76330253ad: Time spent starting tablet: real 0.015s user 0.011s sys 0.002s
I20260924 16:06:02.338054 8640 raft_consensus.cc:482] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:06:02.338534 8640 raft_consensus.cc:504] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc859e16d248a6b2f6ac8e3fc49bcc" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 44231 } } peers { permanent_uuid: "244e4d158ca14d05ae58ad76330253ad" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 45493 } } peers { permanent_uuid: "4d40c54873384912a9dc09622d3aad3c" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 44751 } }
I20260924 16:06:02.340965 8640 leader_election.cc:288] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244e4d158ca14d05ae58ad76330253ad (127.3.214.195:45493), 4d40c54873384912a9dc09622d3aad3c (127.3.214.194:44751)
I20260924 16:06:02.349506 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.341695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:44707 (local address 127.3.214.194:44751)
0924 16:06:02.341954 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:02.341960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:02.343051 (+ 1091us) server_negotiation.cc:407] Connection header received
0924 16:06:02.343155 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:02.343161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:02.343259 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:02.343354 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:02.344285 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.345328 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.347574 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.348459 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.348624 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:02.348704 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:02.349083 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 16:06:02.349220 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260924 16:06:02.349689 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.341553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:44751 (local address 127.3.214.193:44707)
0924 16:06:02.342762 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:02.342782 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:02.342985 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:02.343426 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:02.343432 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:02.343471 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:02.344063 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:02.344082 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.345479 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.345485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:02.347384 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:02.347402 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.348694 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.348701 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:02.348903 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:02.348934 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 16:06:02.349038 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 16:06:02.349196 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1111,"mutex_wait_us":33,"thread_start_us":666,"threads_started":1}
I20260924 16:06:02.349852 8631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.341698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:40529 (local address 127.3.214.195:45493)
0924 16:06:02.341952 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:02.341959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:02.342948 (+ 989us) server_negotiation.cc:407] Connection header received
0924 16:06:02.343240 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:02.343246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:02.343338 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:02.343404 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:02.344464 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.345608 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.347742 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.348774 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.348974 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:02.349065 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:02.349422 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 16:06:02.349568 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 16:06:02.350158 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.341362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:45493 (local address 127.3.214.193:40529)
0924 16:06:02.342737 (+ 1375us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:02.342766 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:02.343036 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:02.343659 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:02.343664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:02.343695 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:02.344260 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:02.344276 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.345777 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.345783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:02.347573 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:02.347589 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.349033 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.349040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:02.349236 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:02.349264 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 16:06:02.349357 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 16:06:02.349532 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1173,"mutex_wait_us":40,"spinlock_wait_cycles":55040,"thread_start_us":894,"threads_started":1}
I20260924 16:06:02.351450 8433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb4a81341434e52ab65dd5de3ab90ba" candidate_uuid: "8dfc859e16d248a6b2f6ac8e3fc49bcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d40c54873384912a9dc09622d3aad3c" is_pre_election: true
I20260924 16:06:02.351742 8568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb4a81341434e52ab65dd5de3ab90ba" candidate_uuid: "8dfc859e16d248a6b2f6ac8e3fc49bcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244e4d158ca14d05ae58ad76330253ad" is_pre_election: true
I20260924 16:06:02.352134 8433 raft_consensus.cc:2454] T 1cb4a81341434e52ab65dd5de3ab90ba P 4d40c54873384912a9dc09622d3aad3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dfc859e16d248a6b2f6ac8e3fc49bcc in term 0.
I20260924 16:06:02.352366 8568 raft_consensus.cc:2454] T 1cb4a81341434e52ab65dd5de3ab90ba P 244e4d158ca14d05ae58ad76330253ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dfc859e16d248a6b2f6ac8e3fc49bcc in term 0.
I20260924 16:06:02.353283 8231 leader_election.cc:302] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244e4d158ca14d05ae58ad76330253ad, 8dfc859e16d248a6b2f6ac8e3fc49bcc; no voters:
I20260924 16:06:02.353775 8640 raft_consensus.cc:2781] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:06:02.353945 8640 raft_consensus.cc:482] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:06:02.354056 8640 raft_consensus.cc:3037] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:02.356752 8640 raft_consensus.cc:504] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc859e16d248a6b2f6ac8e3fc49bcc" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 44231 } } peers { permanent_uuid: "244e4d158ca14d05ae58ad76330253ad" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 45493 } } peers { permanent_uuid: "4d40c54873384912a9dc09622d3aad3c" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 44751 } }
I20260924 16:06:02.357506 8640 leader_election.cc:288] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc [CANDIDATE]: Term 1 election: Requested vote from peers 244e4d158ca14d05ae58ad76330253ad (127.3.214.195:45493), 4d40c54873384912a9dc09622d3aad3c (127.3.214.194:44751)
I20260924 16:06:02.358197 8568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb4a81341434e52ab65dd5de3ab90ba" candidate_uuid: "8dfc859e16d248a6b2f6ac8e3fc49bcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244e4d158ca14d05ae58ad76330253ad"
I20260924 16:06:02.358232 8433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb4a81341434e52ab65dd5de3ab90ba" candidate_uuid: "8dfc859e16d248a6b2f6ac8e3fc49bcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d40c54873384912a9dc09622d3aad3c"
I20260924 16:06:02.358422 8568 raft_consensus.cc:3037] T 1cb4a81341434e52ab65dd5de3ab90ba P 244e4d158ca14d05ae58ad76330253ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:02.358474 8433 raft_consensus.cc:3037] T 1cb4a81341434e52ab65dd5de3ab90ba P 4d40c54873384912a9dc09622d3aad3c [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:02.361131 8433 raft_consensus.cc:2454] T 1cb4a81341434e52ab65dd5de3ab90ba P 4d40c54873384912a9dc09622d3aad3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfc859e16d248a6b2f6ac8e3fc49bcc in term 1.
W20260924 16:06:02.361199 8615 tablet.cc:2369] T 1cb4a81341434e52ab65dd5de3ab90ba P 244e4d158ca14d05ae58ad76330253ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:06:02.361588 8568 raft_consensus.cc:2454] T 1cb4a81341434e52ab65dd5de3ab90ba P 244e4d158ca14d05ae58ad76330253ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfc859e16d248a6b2f6ac8e3fc49bcc in term 1.
I20260924 16:06:02.361750 8233 leader_election.cc:302] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d40c54873384912a9dc09622d3aad3c, 8dfc859e16d248a6b2f6ac8e3fc49bcc; no voters:
I20260924 16:06:02.362202 8640 raft_consensus.cc:2781] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:02.362780 8640 raft_consensus.cc:686] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc [term 1 LEADER]: Becoming Leader. State: Replica: 8dfc859e16d248a6b2f6ac8e3fc49bcc, State: Running, Role: LEADER
I20260924 16:06:02.363447 8640 consensus_queue.cc:237] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc859e16d248a6b2f6ac8e3fc49bcc" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 44231 } } peers { permanent_uuid: "244e4d158ca14d05ae58ad76330253ad" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 45493 } } peers { permanent_uuid: "4d40c54873384912a9dc09622d3aad3c" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 44751 } }
I20260924 16:06:02.369455 8153 catalog_manager.cc:5495] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dfc859e16d248a6b2f6ac8e3fc49bcc (127.3.214.193). New cstate: current_term: 1 leader_uuid: "8dfc859e16d248a6b2f6ac8e3fc49bcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc859e16d248a6b2f6ac8e3fc49bcc" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 44231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "244e4d158ca14d05ae58ad76330253ad" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 45493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d40c54873384912a9dc09622d3aad3c" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 44751 } health_report { overall_health: UNKNOWN } } }
I20260924 16:06:02.400746 8628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.393429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32834 (local address 127.3.214.193:44231)
0924 16:06:02.393655 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:02.393661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:02.393715 (+ 54us) server_negotiation.cc:407] Connection header received
0924 16:06:02.394043 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:02.394047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:02.394204 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:02.394374 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:02.394976 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.395841 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.397956 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.398540 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.398657 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:02.399371 (+ 714us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:02.400065 (+ 694us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:02.400199 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:02.400354 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 16:06:02.400444 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 16:06:02.412763 8433 raft_consensus.cc:1261] T 1cb4a81341434e52ab65dd5de3ab90ba P 4d40c54873384912a9dc09622d3aad3c [term 1 FOLLOWER]: Refusing update from remote peer 8dfc859e16d248a6b2f6ac8e3fc49bcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:02.412763 8568 raft_consensus.cc:1261] T 1cb4a81341434e52ab65dd5de3ab90ba P 244e4d158ca14d05ae58ad76330253ad [term 1 FOLLOWER]: Refusing update from remote peer 8dfc859e16d248a6b2f6ac8e3fc49bcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:02.414191 8640 consensus_queue.cc:1035] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d40c54873384912a9dc09622d3aad3c" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 44751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:02.414600 8644 consensus_queue.cc:1035] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "244e4d158ca14d05ae58ad76330253ad" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 45493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:02.461992 8152 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:02.462361 8152 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:02.467829 8152 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0366fc83bbe490e8271c2bcfeab241f: Sending DeleteTablet for 3 replicas of tablet 1cb4a81341434e52ab65dd5de3ab90ba
I20260924 16:06:02.469455 8413 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cb4a81341434e52ab65dd5de3ab90ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:02 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:40406
I20260924 16:06:02.469653 8278 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cb4a81341434e52ab65dd5de3ab90ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:02 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:32832
I20260924 16:06:02.471207 8548 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cb4a81341434e52ab65dd5de3ab90ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:02 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:60506
I20260924 16:06:02.476346 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 8215
I20260924 16:06:02.486783 8658 tablet_replica.cc:330] T 1cb4a81341434e52ab65dd5de3ab90ba P 4d40c54873384912a9dc09622d3aad3c: stopping tablet replica
I20260924 16:06:02.487433 8658 raft_consensus.cc:2229] T 1cb4a81341434e52ab65dd5de3ab90ba P 4d40c54873384912a9dc09622d3aad3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:06:02.487869 8660 tablet_replica.cc:330] T 1cb4a81341434e52ab65dd5de3ab90ba P 244e4d158ca14d05ae58ad76330253ad: stopping tablet replica
I20260924 16:06:02.488103 8658 raft_consensus.cc:2258] T 1cb4a81341434e52ab65dd5de3ab90ba P 4d40c54873384912a9dc09622d3aad3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:02.488318 8660 raft_consensus.cc:2229] T 1cb4a81341434e52ab65dd5de3ab90ba P 244e4d158ca14d05ae58ad76330253ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:06:02.489013 8659 tablet_replica.cc:330] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc: stopping tablet replica
I20260924 16:06:02.489532 8660 raft_consensus.cc:2258] T 1cb4a81341434e52ab65dd5de3ab90ba P 244e4d158ca14d05ae58ad76330253ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:02.490588 8659 raft_consensus.cc:2229] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:06:02.490923 8658 ts_tablet_manager.cc:1905] T 1cb4a81341434e52ab65dd5de3ab90ba P 4d40c54873384912a9dc09622d3aad3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:02.492046 8660 ts_tablet_manager.cc:1905] T 1cb4a81341434e52ab65dd5de3ab90ba P 244e4d158ca14d05ae58ad76330253ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:02.496821 8659 raft_consensus.cc:2258] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:02.498344 8658 ts_tablet_manager.cc:1918] T 1cb4a81341434e52ab65dd5de3ab90ba P 4d40c54873384912a9dc09622d3aad3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:06:02.499274 8660 ts_tablet_manager.cc:1918] T 1cb4a81341434e52ab65dd5de3ab90ba P 244e4d158ca14d05ae58ad76330253ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:06:02.499567 8660 log.cc:1196] T 1cb4a81341434e52ab65dd5de3ab90ba P 244e4d158ca14d05ae58ad76330253ad: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/1cb4a81341434e52ab65dd5de3ab90ba
I20260924 16:06:02.500214 8660 ts_tablet_manager.cc:1939] T 1cb4a81341434e52ab65dd5de3ab90ba P 244e4d158ca14d05ae58ad76330253ad: Deleting consensus metadata
Sep 24 16:06:02 dist-test-slave-1gc3 krb5kdc[8109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.193@KRBTEST.COM
I20260924 16:06:02.502689 8138 catalog_manager.cc:4841] TS 244e4d158ca14d05ae58ad76330253ad (127.3.214.195:45493): tablet 1cb4a81341434e52ab65dd5de3ab90ba (table test-table [id=05b2bb24a4674e508184d6886e4c56f6]) successfully deleted
I20260924 16:06:02.502938 8659 ts_tablet_manager.cc:1905] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:02.503485 8658 log.cc:1196] T 1cb4a81341434e52ab65dd5de3ab90ba P 4d40c54873384912a9dc09622d3aad3c: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/1cb4a81341434e52ab65dd5de3ab90ba
I20260924 16:06:02.504225 8658 ts_tablet_manager.cc:1939] T 1cb4a81341434e52ab65dd5de3ab90ba P 4d40c54873384912a9dc09622d3aad3c: Deleting consensus metadata
I20260924 16:06:02.506657 8140 catalog_manager.cc:4841] TS 4d40c54873384912a9dc09622d3aad3c (127.3.214.194:44751): tablet 1cb4a81341434e52ab65dd5de3ab90ba (table test-table [id=05b2bb24a4674e508184d6886e4c56f6]) successfully deleted
I20260924 16:06:02.521369 8628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.478381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32846 (local address 127.3.214.193:44231)
0924 16:06:02.478607 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:02.478614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:02.487198 (+ 8584us) server_negotiation.cc:407] Connection header received
0924 16:06:02.487424 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:02.487431 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:02.487553 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:02.487731 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:02.490696 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.492013 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.493998 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.494789 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.494906 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:02.511687 (+ 16781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:02.511710 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:02.511735 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:02.511834 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:02.516405 (+ 4571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:02.517553 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:02.517564 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:02.517568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:02.517772 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:02.518250 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:02.518255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:02.518259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:02.518812 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:02.519157 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:02.520626 (+ 1469us) server_negotiation.cc:299] Negotiation successful
0924 16:06:02.520926 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 16:06:02.522809 8338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:02.703337 8659 ts_tablet_manager.cc:1918] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:06:02.703753 8659 log.cc:1196] T 1cb4a81341434e52ab65dd5de3ab90ba P 8dfc859e16d248a6b2f6ac8e3fc49bcc: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790265920616583-3931-0/minicluster-data/ts-0/wal/wals/1cb4a81341434e52ab65dd5de3ab90ba
I20260924 16:06:02.704391 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 8215
I20260924 16:06:02.725888 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 8350
W20260924 16:06:02.725929 8138 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.214.193:44231 (error 108)
I20260924 16:06:02.728142 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.727502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:44231 (local address 127.0.0.1:32850)
0924 16:06:02.727757 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:02.727899 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:44231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260924 16:06:02.728514 8138 catalog_manager.cc:4584] TS 8dfc859e16d248a6b2f6ac8e3fc49bcc (127.3.214.193:44231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1cb4a81341434e52ab65dd5de3ab90ba: Network error: Client connection negotiation failed: client connection to 127.3.214.193:44231: connect: Connection refused (error 111)
Sep 24 16:06:02 dist-test-slave-1gc3 krb5kdc[8109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.194@KRBTEST.COM
I20260924 16:06:02.743976 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.727024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40412 (local address 127.3.214.194:44751)
0924 16:06:02.727207 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:02.727211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:02.727235 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:06:02.727477 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:02.727483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:02.727581 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:02.727705 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:02.729343 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.730209 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.731554 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.731970 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.732070 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:02.736922 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:02.736945 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:02.736967 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:02.737033 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:02.740272 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:02.741425 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:02.741435 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:02.741439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:02.741618 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:02.742091 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:02.742134 (+ 43us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:02.742138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:02.742639 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:02.742918 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:02.743386 (+ 468us) server_negotiation.cc:299] Negotiation successful
0924 16:06:02.743605 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:06:02.744704 8473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:02.782923 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.782003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:44231 (local address 127.0.0.1:32864)
0924 16:06:02.782434 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:02.782623 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:44231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
I20260924 16:06:02.784384 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.783899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:44231 (local address 127.0.0.1:32868)
0924 16:06:02.784072 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:02.784204 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:44231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 16:06:02.845259 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.844495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:44231 (local address 127.0.0.1:32884)
0924 16:06:02.844893 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:02.845023 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:44231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205}
I20260924 16:06:02.846591 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 8350
I20260924 16:06:02.846769 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.846362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:44231 (local address 127.0.0.1:32886)
0924 16:06:02.846541 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:02.846623 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:44231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 16:06:02.867502 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 8484
Sep 24 16:06:02 dist-test-slave-1gc3 krb5kdc[8109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.195@KRBTEST.COM
I20260924 16:06:02.885648 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.868572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60516 (local address 127.3.214.195:45493)
0924 16:06:02.869152 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:02.869158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:02.869180 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:06:02.869278 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:02.869282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:02.869358 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:02.869486 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:02.871076 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.871908 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.873305 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:02.873716 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:02.873826 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:02.879081 (+ 5255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:02.879106 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:02.879134 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:02.879184 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:02.882345 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:02.883277 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:02.883286 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:02.883289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:02.883458 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:02.883875 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:02.883880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:02.883883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:02.884386 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:02.884715 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:02.885017 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 16:06:02.885237 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":330,"threads_started":1}
I20260924 16:06:02.886626 8609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:02.929131 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.928451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:44231 (local address 127.0.0.1:32888)
0924 16:06:02.928791 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:02.928924 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:44231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260924 16:06:02.930718 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:02.930091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:44231 (local address 127.0.0.1:32902)
0924 16:06:02.930446 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:02.930570 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:44231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260924 16:06:02.992614 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 8484
I20260924 16:06:03.012477 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 8122
I20260924 16:06:03.014042 8184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:03.130666 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 8122
2026-09-24T16:06:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4692 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T16:06:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:03Z Disabled control of system clock
I20260924 16:06:03.206923 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:39257
--webserver_interface=127.3.214.254
--webserver_port=0
--builtin_ntp_servers=127.3.214.212:43641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:39257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 16:06:03.578821 8671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:03.579185 8671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:03.579298 8671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:03.587373 8671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:03.587539 8671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:03.587580 8671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:03.587685 8671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:03.587755 8671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:03.598487 8671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:43641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:39257
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:39257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:03.600025 8671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:03.601954 8671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:03.616212 8679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:03.616231 8677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:03.616283 8676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:03.617647 8671 server_base.cc:1034] running on GCE node
I20260924 16:06:03.618669 8671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:03.620401 8671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:03.621660 8671 hybrid_clock.cc:648] HybridClock initialized: now 1790265963621600 us; error 58 us; skew 500 ppm
I20260924 16:06:03.622092 8671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:06:03.625053 8671 webserver.cc:460] Webserver started at http://127.3.214.254:39129/ using document root <none> and password file <none>
I20260924 16:06:03.625814 8671 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:03.625931 8671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:03.626302 8671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:03.629529 8671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "22c841da0d02445597414e46e71c270d"
format_stamp: "Formatted at 2026-09-24 16:06:03 on dist-test-slave-1gc3"
server_key: "69e14f549e2463a1d7e8babf844e2361"
server_key_iv: "489f150d7d6b1ec5e1c6a5795a50cfd1"
server_key_version: "encryptionkey@0"
I20260924 16:06:03.630442 8671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "22c841da0d02445597414e46e71c270d"
format_stamp: "Formatted at 2026-09-24 16:06:03 on dist-test-slave-1gc3"
server_key: "69e14f549e2463a1d7e8babf844e2361"
server_key_iv: "489f150d7d6b1ec5e1c6a5795a50cfd1"
server_key_version: "encryptionkey@0"
I20260924 16:06:03.636705 8671 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 16:06:03.641008 8685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:03.642915 8671 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:06:03.643155 8671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "22c841da0d02445597414e46e71c270d"
format_stamp: "Formatted at 2026-09-24 16:06:03 on dist-test-slave-1gc3"
server_key: "69e14f549e2463a1d7e8babf844e2361"
server_key_iv: "489f150d7d6b1ec5e1c6a5795a50cfd1"
server_key_version: "encryptionkey@0"
I20260924 16:06:03.643397 8671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:03.650231 8671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:03.651221 8671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:03.651528 8671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:03.674508 8671 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:39257
I20260924 16:06:03.674541 8737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:39257 every 8 connection(s)
I20260924 16:06:03.676416 8671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:03.680989 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 8671
I20260924 16:06:03.681187 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:06:03.681954 3931 external_mini_cluster.cc:1428] Setting key 43cb657eb40e498bfdc29095ae64094b
I20260924 16:06:03.681963 8738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:03.692579 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:03.684442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46822 (local address 127.3.214.254:39257)
0924 16:06:03.686014 (+ 1572us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:03.686029 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:03.686138 (+ 109us) server_negotiation.cc:407] Connection header received
0924 16:06:03.688140 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:03.688171 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:03.688766 (+ 595us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:03.689233 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:03.690611 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:03.690618 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:03.690633 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:03.690654 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:03.690725 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:03.690746 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:03.691035 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:03.691196 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 16:06:03.691525 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":538,"threads_started":1}
I20260924 16:06:03.698966 8738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d: Bootstrap starting.
I20260924 16:06:03.703039 8738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:03.704330 8738 log.cc:824] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:03.708238 8738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d: No bootstrap required, opened a new log
I20260924 16:06:03.717072 8738 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c841da0d02445597414e46e71c270d" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39257 } }
I20260924 16:06:03.717623 8738 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:03.717731 8738 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22c841da0d02445597414e46e71c270d, State: Initialized, Role: FOLLOWER
I20260924 16:06:03.718575 8738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c841da0d02445597414e46e71c270d" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39257 } }
I20260924 16:06:03.718873 8738 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:03.719019 8738 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:03.719200 8738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:03.721792 8738 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c841da0d02445597414e46e71c270d" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39257 } }
I20260924 16:06:03.722520 8738 leader_election.cc:302] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22c841da0d02445597414e46e71c270d; no voters:
I20260924 16:06:03.723354 8738 leader_election.cc:288] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:03.723628 8743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:03.724996 8743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d [term 1 LEADER]: Becoming Leader. State: Replica: 22c841da0d02445597414e46e71c270d, State: Running, Role: LEADER
I20260924 16:06:03.725728 8743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c841da0d02445597414e46e71c270d" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39257 } }
I20260924 16:06:03.726248 8738 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:03.728752 8745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22c841da0d02445597414e46e71c270d. Latest consensus state: current_term: 1 leader_uuid: "22c841da0d02445597414e46e71c270d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c841da0d02445597414e46e71c270d" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39257 } } }
I20260924 16:06:03.729153 8745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:03.729650 8744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22c841da0d02445597414e46e71c270d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c841da0d02445597414e46e71c270d" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39257 } } }
I20260924 16:06:03.730033 8744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:03.732718 8749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:03.739252 8749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:03.750986 8749 catalog_manager.cc:1347] Generated new cluster ID: 0dd7cc853cde4bac9a6a44169f2bbee3
I20260924 16:06:03.751137 8749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:03.780133 8749 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:03.780768 8749 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:03.794723 8749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d: Generated new TSK 0
I20260924 16:06:03.796334 8749 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:03.803508 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39257
--builtin_ntp_servers=127.3.214.212:43641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 16:06:04.174084 8762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:04.174512 8762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:04.174633 8762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:04.182524 8762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:04.182667 8762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:04.182793 8762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:06:04.192885 8762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:43641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:04.194478 8762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:04.196595 8762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:04.212448 8767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:04.212507 8770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:04.212405 8768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:04.216094 8762 server_base.cc:1034] running on GCE node
I20260924 16:06:04.216877 8762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:04.218004 8762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:04.219242 8762 hybrid_clock.cc:648] HybridClock initialized: now 1790265964219182 us; error 64 us; skew 500 ppm
I20260924 16:06:04.219615 8762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:06:04.222208 8762 webserver.cc:460] Webserver started at http://127.3.214.193:40957/ using document root <none> and password file <none>
I20260924 16:06:04.222950 8762 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:04.223048 8762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:04.223371 8762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:04.226531 8762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "94ca0b0c45ce4b11a232ca27399e4901"
format_stamp: "Formatted at 2026-09-24 16:06:04 on dist-test-slave-1gc3"
server_key: "758c135e53bdb818d21a627f237a9dc8"
server_key_iv: "f41b582cfbf99e3c89db918a8967b6ed"
server_key_version: "encryptionkey@0"
I20260924 16:06:04.227345 8762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "94ca0b0c45ce4b11a232ca27399e4901"
format_stamp: "Formatted at 2026-09-24 16:06:04 on dist-test-slave-1gc3"
server_key: "758c135e53bdb818d21a627f237a9dc8"
server_key_iv: "f41b582cfbf99e3c89db918a8967b6ed"
server_key_version: "encryptionkey@0"
I20260924 16:06:04.233314 8762 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260924 16:06:04.237859 8776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:04.239771 8762 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:06:04.240017 8762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "94ca0b0c45ce4b11a232ca27399e4901"
format_stamp: "Formatted at 2026-09-24 16:06:04 on dist-test-slave-1gc3"
server_key: "758c135e53bdb818d21a627f237a9dc8"
server_key_iv: "f41b582cfbf99e3c89db918a8967b6ed"
server_key_version: "encryptionkey@0"
I20260924 16:06:04.240289 8762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:04.247004 8762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:04.247988 8762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:04.248277 8762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:04.249483 8762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:04.251228 8762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:04.251335 8762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:04.251453 8762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:04.251520 8762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:04.288764 8762 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:37955
I20260924 16:06:04.288880 8890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:37955 every 8 connection(s)
I20260924 16:06:04.290380 8762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:06:04.297092 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 8762
I20260924 16:06:04.297335 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:06:04.298080 3931 external_mini_cluster.cc:1428] Setting key 5fa6397479979232f83048550950b7e2
I20260924 16:06:04.301164 8893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.294085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:52497 (local address 127.3.214.254:39257)
0924 16:06:04.294702 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:04.294707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:04.296306 (+ 1599us) server_negotiation.cc:407] Connection header received
0924 16:06:04.298423 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:04.298435 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:04.298517 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:04.298619 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:04.300213 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:04.300221 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:04.300227 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:04.300254 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:04.300324 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:04.300342 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:04.300456 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:04.300729 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 16:06:04.300807 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":306,"threads_started":1}
I20260924 16:06:04.301716 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39257
--builtin_ntp_servers=127.3.214.212:43641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 16:06:04.302168 8894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.294852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39257 (local address 127.3.214.193:52497)
0924 16:06:04.296090 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:04.296144 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:04.297964 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:04.298898 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:04.298908 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:04.299033 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:04.299804 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:06:04.299808 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:04.299863 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:06:04.299996 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:04.300458 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:04.300469 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:04.300478 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 16:06:04.300760 (+ 282us) client_negotiation.cc:240] Negotiation successful
0924 16:06:04.301019 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":344,"threads_started":1}
I20260924 16:06:04.303988 8891 heartbeater.cc:347] Connected to a master server at 127.3.214.254:39257
I20260924 16:06:04.304468 8891 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:04.305409 8891 heartbeater.cc:511] Master 127.3.214.254:39257 requested a full tablet report, sending...
I20260924 16:06:04.308192 8703 ts_manager.cc:194] Registered new tserver with Master: 94ca0b0c45ce4b11a232ca27399e4901 (127.3.214.193:37955)
W20260924 16:06:04.708411 8895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:04.708912 8895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:04.709064 8895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:04.717504 8895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:04.717732 8895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:04.717916 8895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:06:04.729001 8895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:43641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:04.731096 8895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:04.733717 8895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:04.750517 8901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:04.750521 8903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:04.750556 8900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:04.751802 8895 server_base.cc:1034] running on GCE node
I20260924 16:06:04.752646 8895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:04.753975 8895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:04.755225 8895 hybrid_clock.cc:648] HybridClock initialized: now 1790265964755147 us; error 64 us; skew 500 ppm
I20260924 16:06:04.755656 8895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:06:04.758921 8895 webserver.cc:460] Webserver started at http://127.3.214.194:42003/ using document root <none> and password file <none>
I20260924 16:06:04.759997 8895 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:04.760138 8895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:04.760541 8895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:04.764366 8895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "cab721d236394e539176764570e64ed7"
format_stamp: "Formatted at 2026-09-24 16:06:04 on dist-test-slave-1gc3"
server_key: "6d766dbcc8f9b4b366a68821f15f0e22"
server_key_iv: "a4e63800701d1b5c4ec74d3d780c9864"
server_key_version: "encryptionkey@0"
I20260924 16:06:04.765395 8895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "cab721d236394e539176764570e64ed7"
format_stamp: "Formatted at 2026-09-24 16:06:04 on dist-test-slave-1gc3"
server_key: "6d766dbcc8f9b4b366a68821f15f0e22"
server_key_iv: "a4e63800701d1b5c4ec74d3d780c9864"
server_key_version: "encryptionkey@0"
I20260924 16:06:04.772259 8895 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260924 16:06:04.776996 8909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:04.779105 8895 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:06:04.779341 8895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "cab721d236394e539176764570e64ed7"
format_stamp: "Formatted at 2026-09-24 16:06:04 on dist-test-slave-1gc3"
server_key: "6d766dbcc8f9b4b366a68821f15f0e22"
server_key_iv: "a4e63800701d1b5c4ec74d3d780c9864"
server_key_version: "encryptionkey@0"
I20260924 16:06:04.779664 8895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:04.787181 8895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:04.788321 8895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:04.788635 8895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:04.789866 8895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:04.791811 8895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:04.791921 8895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:04.792050 8895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:04.792150 8895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:04.831080 8895 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:44407
I20260924 16:06:04.831190 9024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:44407 every 8 connection(s)
I20260924 16:06:04.832727 8895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:06:04.841629 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 8895
I20260924 16:06:04.841821 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:06:04.842437 3931 external_mini_cluster.cc:1428] Setting key 475c4796e2d39e994c8ca20bdb752408
I20260924 16:06:04.843006 9027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.836009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:47763 (local address 127.3.214.254:39257)
0924 16:06:04.836583 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:04.836589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:04.838277 (+ 1688us) server_negotiation.cc:407] Connection header received
0924 16:06:04.840227 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:04.840233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:04.840311 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:04.840409 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:04.842088 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:04.842115 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:04.842121 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:04.842152 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:04.842218 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:04.842235 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:04.842402 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:04.842642 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 16:06:04.842714 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":281,"threads_started":1}
I20260924 16:06:04.844310 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:04.836835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39257 (local address 127.3.214.194:47763)
0924 16:06:04.838044 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:04.838132 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:04.839919 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:04.840693 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:04.840702 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:04.840809 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:04.841610 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:06:04.841614 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:04.841670 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:06:04.841803 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:04.842385 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:04.842398 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:04.842407 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 16:06:04.842755 (+ 348us) client_negotiation.cc:240] Negotiation successful
0924 16:06:04.843096 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":330,"threads_started":1}
I20260924 16:06:04.846172 9025 heartbeater.cc:347] Connected to a master server at 127.3.214.254:39257
I20260924 16:06:04.846289 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39257
--builtin_ntp_servers=127.3.214.212:43641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 16:06:04.846575 9025 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:04.847266 9025 heartbeater.cc:511] Master 127.3.214.254:39257 requested a full tablet report, sending...
I20260924 16:06:04.848994 8703 ts_manager.cc:194] Registered new tserver with Master: cab721d236394e539176764570e64ed7 (127.3.214.194:44407)
W20260924 16:06:05.230882 9029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:05.231205 9029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:05.231304 9029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:05.239102 9029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:05.239228 9029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:05.239342 9029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:06:05.249119 9029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:43641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:05.251117 9029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:05.253017 9029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:05.268127 9035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:05.268460 9037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:05.268947 9034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:05.269155 9029 server_base.cc:1034] running on GCE node
I20260924 16:06:05.269893 9029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:05.271004 9029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:05.272251 9029 hybrid_clock.cc:648] HybridClock initialized: now 1790265965272169 us; error 69 us; skew 500 ppm
I20260924 16:06:05.272639 9029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:06:05.275341 9029 webserver.cc:460] Webserver started at http://127.3.214.195:38679/ using document root <none> and password file <none>
I20260924 16:06:05.276125 9029 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:05.276230 9029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:05.276569 9029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:05.279671 9029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "da58c42b1be346a69ca0b7a65e03def6"
format_stamp: "Formatted at 2026-09-24 16:06:05 on dist-test-slave-1gc3"
server_key: "34f6589581e7dca3c298320b1a2000af"
server_key_iv: "85507837c32b607cc62e9a76a4250db2"
server_key_version: "encryptionkey@0"
I20260924 16:06:05.280508 9029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "da58c42b1be346a69ca0b7a65e03def6"
format_stamp: "Formatted at 2026-09-24 16:06:05 on dist-test-slave-1gc3"
server_key: "34f6589581e7dca3c298320b1a2000af"
server_key_iv: "85507837c32b607cc62e9a76a4250db2"
server_key_version: "encryptionkey@0"
I20260924 16:06:05.286666 9029 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 16:06:05.291572 9043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:05.294052 9029 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20260924 16:06:05.294354 9029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "da58c42b1be346a69ca0b7a65e03def6"
format_stamp: "Formatted at 2026-09-24 16:06:05 on dist-test-slave-1gc3"
server_key: "34f6589581e7dca3c298320b1a2000af"
server_key_iv: "85507837c32b607cc62e9a76a4250db2"
server_key_version: "encryptionkey@0"
I20260924 16:06:05.294642 9029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:05.301285 9029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:05.302301 9029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:05.302591 9029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:05.303643 9029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:05.305289 9029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:05.305400 9029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:05.305519 9029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:05.305599 9029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:05.312527 8891 heartbeater.cc:503] Master 127.3.214.254:39257 was elected leader, sending a full tablet report...
I20260924 16:06:05.342669 9029 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:42671
I20260924 16:06:05.342784 9157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:42671 every 8 connection(s)
I20260924 16:06:05.344141 9029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:06:05.353396 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 9029
I20260924 16:06:05.353605 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:06:05.354195 3931 external_mini_cluster.cc:1428] Setting key 1edc72bfabcdf689e8b21821300a2a85
I20260924 16:06:05.354805 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.347561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:55617 (local address 127.3.214.254:39257)
0924 16:06:05.348181 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:05.348187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:05.349735 (+ 1548us) server_negotiation.cc:407] Connection header received
0924 16:06:05.351710 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:05.351716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:05.351790 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:05.351887 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:05.353653 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:05.353659 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:05.353664 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:05.353688 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:05.353749 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:05.353768 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:05.353925 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:05.354301 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 16:06:05.354425 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":284,"threads_started":1}
I20260924 16:06:05.356213 9161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.348377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39257 (local address 127.3.214.195:55617)
0924 16:06:05.349535 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:05.349602 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:05.351389 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:05.352172 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:05.352181 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:05.352291 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:05.353182 (+ 891us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:06:05.353186 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:05.353244 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:06:05.353385 (+ 141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:05.353943 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:05.353958 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:05.353969 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 16:06:05.354412 (+ 443us) client_negotiation.cc:240] Negotiation successful
0924 16:06:05.354746 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":313,"threads_started":1}
I20260924 16:06:05.358202 9158 heartbeater.cc:347] Connected to a master server at 127.3.214.254:39257
I20260924 16:06:05.358620 9158 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:05.359329 9158 heartbeater.cc:511] Master 127.3.214.254:39257 requested a full tablet report, sending...
I20260924 16:06:05.361232 8703 ts_manager.cc:194] Registered new tserver with Master: da58c42b1be346a69ca0b7a65e03def6 (127.3.214.195:42671)
I20260924 16:06:05.372165 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:06:05.380352 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.377670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38712 (local address 127.3.214.254:39257)
0924 16:06:05.377923 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:05.377931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:05.378457 (+ 526us) server_negotiation.cc:407] Connection header received
0924 16:06:05.378642 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:05.378646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:05.378708 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:05.378806 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:05.379815 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:05.379819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:05.379822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:05.379838 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:05.379873 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:05.379885 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:05.380002 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:05.380041 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 16:06:05.380087 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 16:06:05.385270 8703 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:06:05.393028 8703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:06:05.416602 9171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.409231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:44407 (local address 127.0.0.1:34320)
0924 16:06:05.411065 (+ 1834us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:05.411112 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:05.411296 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:05.412757 (+ 1461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:05.412766 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:05.412877 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:05.413808 (+ 931us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:06:05.413812 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:05.413845 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:06:05.413943 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:05.415704 (+ 1761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:05.415719 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:05.415729 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 16:06:05.415924 (+ 195us) client_negotiation.cc:240] Negotiation successful
0924 16:06:05.416058 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1540,"thread_start_us":484,"threads_started":1}
I20260924 16:06:05.417748 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.409933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:42671 (local address 127.0.0.1:55630)
0924 16:06:05.412659 (+ 2726us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:05.412683 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:05.412829 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:05.414965 (+ 2136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:05.414970 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:05.414995 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:05.415812 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:06:05.415816 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:05.415842 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:06:05.415901 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:05.416558 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:05.416564 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:05.416570 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 16:06:05.417100 (+ 530us) client_negotiation.cc:240] Negotiation successful
0924 16:06:05.417176 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2548,"thread_start_us":2459,"threads_started":1}
I20260924 16:06:05.417870 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.409462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34320 (local address 127.3.214.194:44407)
0924 16:06:05.412165 (+ 2703us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:05.412171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:05.412199 (+ 28us) server_negotiation.cc:407] Connection header received
0924 16:06:05.412261 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:05.412267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:05.412525 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:05.413363 (+ 838us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:05.415427 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:05.415437 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:05.415454 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:05.415478 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:05.415538 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:05.415558 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:05.417261 (+ 1703us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:05.417352 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 16:06:05.417480 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2548,"thread_start_us":2400,"threads_started":1}
I20260924 16:06:05.419478 9174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.410288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55630 (local address 127.3.214.195:42671)
0924 16:06:05.411872 (+ 1584us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:05.411882 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:05.412996 (+ 1114us) server_negotiation.cc:407] Connection header received
0924 16:06:05.413117 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:05.413128 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:05.413403 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:05.413589 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:05.416129 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:05.416140 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:05.416158 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:05.416194 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:05.416269 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:05.416295 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:05.416497 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:05.418840 (+ 2343us) server_negotiation.cc:299] Negotiation successful
0924 16:06:05.418984 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1423,"thread_start_us":1257,"threads_started":1}
I20260924 16:06:05.420425 9171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.412635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:37955 (local address 127.0.0.1:33154)
0924 16:06:05.416887 (+ 4252us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:05.416910 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:05.417038 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:05.417749 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:05.417755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:05.417807 (+ 52us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:05.418465 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:06:05.418469 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:05.418494 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:06:05.418556 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:05.419961 (+ 1405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:05.419967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:05.419973 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 16:06:05.420070 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 16:06:05.420135 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4126,"thread_start_us":5787,"threads_started":1}
I20260924 16:06:05.421487 9176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.413219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.3.214.193:37955)
0924 16:06:05.414322 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:05.414334 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:05.417053 (+ 2719us) server_negotiation.cc:407] Connection header received
0924 16:06:05.417175 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:05.417186 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:05.417484 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:05.418783 (+ 1299us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:05.419696 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:05.419706 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:05.419722 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:05.419739 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:05.419796 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:05.419817 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:05.420944 (+ 1127us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:05.421033 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 16:06:05.421150 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":620,"threads_started":1}
I20260924 16:06:05.423493 8959 tablet_service.cc:1467] Processing CreateTablet for tablet 10d1458c0547468cbe8b89750ca59506 (DEFAULT_TABLE table=test-table [id=2deb1de666694835ad789cbc04c99a30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:05.423906 9092 tablet_service.cc:1467] Processing CreateTablet for tablet 10d1458c0547468cbe8b89750ca59506 (DEFAULT_TABLE table=test-table [id=2deb1de666694835ad789cbc04c99a30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:05.425014 8959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10d1458c0547468cbe8b89750ca59506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:05.426231 9092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10d1458c0547468cbe8b89750ca59506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:05.425885 8825 tablet_service.cc:1467] Processing CreateTablet for tablet 10d1458c0547468cbe8b89750ca59506 (DEFAULT_TABLE table=test-table [id=2deb1de666694835ad789cbc04c99a30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:05.427927 8825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10d1458c0547468cbe8b89750ca59506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:05.438372 9177 tablet_bootstrap.cc:492] T 10d1458c0547468cbe8b89750ca59506 P cab721d236394e539176764570e64ed7: Bootstrap starting.
I20260924 16:06:05.441947 9177 tablet_bootstrap.cc:654] T 10d1458c0547468cbe8b89750ca59506 P cab721d236394e539176764570e64ed7: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:05.443423 9177 log.cc:824] T 10d1458c0547468cbe8b89750ca59506 P cab721d236394e539176764570e64ed7: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:05.445323 9178 tablet_bootstrap.cc:492] T 10d1458c0547468cbe8b89750ca59506 P da58c42b1be346a69ca0b7a65e03def6: Bootstrap starting.
I20260924 16:06:05.446009 9179 tablet_bootstrap.cc:492] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901: Bootstrap starting.
I20260924 16:06:05.449509 9177 tablet_bootstrap.cc:492] T 10d1458c0547468cbe8b89750ca59506 P cab721d236394e539176764570e64ed7: No bootstrap required, opened a new log
I20260924 16:06:05.449908 9177 ts_tablet_manager.cc:1397] T 10d1458c0547468cbe8b89750ca59506 P cab721d236394e539176764570e64ed7: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260924 16:06:05.450083 9178 tablet_bootstrap.cc:654] T 10d1458c0547468cbe8b89750ca59506 P da58c42b1be346a69ca0b7a65e03def6: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:05.450543 9179 tablet_bootstrap.cc:654] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:05.451530 9178 log.cc:824] T 10d1458c0547468cbe8b89750ca59506 P da58c42b1be346a69ca0b7a65e03def6: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:05.451670 9179 log.cc:824] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:05.455395 9178 tablet_bootstrap.cc:492] T 10d1458c0547468cbe8b89750ca59506 P da58c42b1be346a69ca0b7a65e03def6: No bootstrap required, opened a new log
I20260924 16:06:05.455693 9178 ts_tablet_manager.cc:1397] T 10d1458c0547468cbe8b89750ca59506 P da58c42b1be346a69ca0b7a65e03def6: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.001s
I20260924 16:06:05.456538 9179 tablet_bootstrap.cc:492] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901: No bootstrap required, opened a new log
I20260924 16:06:05.456928 9179 ts_tablet_manager.cc:1397] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20260924 16:06:05.462432 9177 raft_consensus.cc:348] T 10d1458c0547468cbe8b89750ca59506 P cab721d236394e539176764570e64ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab721d236394e539176764570e64ed7" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 44407 } } peers { permanent_uuid: "da58c42b1be346a69ca0b7a65e03def6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42671 } } peers { permanent_uuid: "94ca0b0c45ce4b11a232ca27399e4901" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37955 } }
I20260924 16:06:05.463229 9177 raft_consensus.cc:374] T 10d1458c0547468cbe8b89750ca59506 P cab721d236394e539176764570e64ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:05.463456 9177 raft_consensus.cc:729] T 10d1458c0547468cbe8b89750ca59506 P cab721d236394e539176764570e64ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab721d236394e539176764570e64ed7, State: Initialized, Role: FOLLOWER
I20260924 16:06:05.464504 9177 consensus_queue.cc:260] T 10d1458c0547468cbe8b89750ca59506 P cab721d236394e539176764570e64ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab721d236394e539176764570e64ed7" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 44407 } } peers { permanent_uuid: "da58c42b1be346a69ca0b7a65e03def6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42671 } } peers { permanent_uuid: "94ca0b0c45ce4b11a232ca27399e4901" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37955 } }
I20260924 16:06:05.464525 9178 raft_consensus.cc:348] T 10d1458c0547468cbe8b89750ca59506 P da58c42b1be346a69ca0b7a65e03def6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab721d236394e539176764570e64ed7" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 44407 } } peers { permanent_uuid: "da58c42b1be346a69ca0b7a65e03def6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42671 } } peers { permanent_uuid: "94ca0b0c45ce4b11a232ca27399e4901" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37955 } }
I20260924 16:06:05.465191 9178 raft_consensus.cc:374] T 10d1458c0547468cbe8b89750ca59506 P da58c42b1be346a69ca0b7a65e03def6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:05.465318 9178 raft_consensus.cc:729] T 10d1458c0547468cbe8b89750ca59506 P da58c42b1be346a69ca0b7a65e03def6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da58c42b1be346a69ca0b7a65e03def6, State: Initialized, Role: FOLLOWER
I20260924 16:06:05.466367 9178 consensus_queue.cc:260] T 10d1458c0547468cbe8b89750ca59506 P da58c42b1be346a69ca0b7a65e03def6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab721d236394e539176764570e64ed7" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 44407 } } peers { permanent_uuid: "da58c42b1be346a69ca0b7a65e03def6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42671 } } peers { permanent_uuid: "94ca0b0c45ce4b11a232ca27399e4901" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37955 } }
I20260924 16:06:05.467322 9025 heartbeater.cc:503] Master 127.3.214.254:39257 was elected leader, sending a full tablet report...
I20260924 16:06:05.468220 9177 ts_tablet_manager.cc:1428] T 10d1458c0547468cbe8b89750ca59506 P cab721d236394e539176764570e64ed7: Time spent starting tablet: real 0.018s user 0.010s sys 0.009s
I20260924 16:06:05.469424 9179 raft_consensus.cc:348] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab721d236394e539176764570e64ed7" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 44407 } } peers { permanent_uuid: "da58c42b1be346a69ca0b7a65e03def6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42671 } } peers { permanent_uuid: "94ca0b0c45ce4b11a232ca27399e4901" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37955 } }
I20260924 16:06:05.469885 9179 raft_consensus.cc:374] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:05.470021 9179 raft_consensus.cc:729] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ca0b0c45ce4b11a232ca27399e4901, State: Initialized, Role: FOLLOWER
I20260924 16:06:05.470775 9179 consensus_queue.cc:260] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab721d236394e539176764570e64ed7" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 44407 } } peers { permanent_uuid: "da58c42b1be346a69ca0b7a65e03def6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42671 } } peers { permanent_uuid: "94ca0b0c45ce4b11a232ca27399e4901" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37955 } }
I20260924 16:06:05.472666 9179 ts_tablet_manager.cc:1428] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901: Time spent starting tablet: real 0.016s user 0.011s sys 0.006s
I20260924 16:06:05.474026 9158 heartbeater.cc:503] Master 127.3.214.254:39257 was elected leader, sending a full tablet report...
I20260924 16:06:05.475378 9178 ts_tablet_manager.cc:1428] T 10d1458c0547468cbe8b89750ca59506 P da58c42b1be346a69ca0b7a65e03def6: Time spent starting tablet: real 0.020s user 0.007s sys 0.012s
W20260924 16:06:05.543308 8892 tablet.cc:2369] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:06:05.585258 9026 tablet.cc:2369] T 10d1458c0547468cbe8b89750ca59506 P cab721d236394e539176764570e64ed7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:06:05.596599 9159 tablet.cc:2369] T 10d1458c0547468cbe8b89750ca59506 P da58c42b1be346a69ca0b7a65e03def6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:06:05.799048 9185 raft_consensus.cc:482] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:06:05.799461 9185 raft_consensus.cc:504] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab721d236394e539176764570e64ed7" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 44407 } } peers { permanent_uuid: "da58c42b1be346a69ca0b7a65e03def6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42671 } } peers { permanent_uuid: "94ca0b0c45ce4b11a232ca27399e4901" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37955 } }
I20260924 16:06:05.801404 9185 leader_election.cc:288] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab721d236394e539176764570e64ed7 (127.3.214.194:44407), da58c42b1be346a69ca0b7a65e03def6 (127.3.214.195:42671)
I20260924 16:06:05.805655 9174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.802189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:60137 (local address 127.3.214.195:42671)
0924 16:06:05.802430 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:05.802438 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:05.803348 (+ 910us) server_negotiation.cc:407] Connection header received
0924 16:06:05.803535 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:05.803539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:05.803606 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:05.803701 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:05.804962 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:05.804968 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:05.804971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:05.804994 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:05.805038 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:05.805052 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:05.805190 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:05.805268 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 16:06:05.805347 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 16:06:05.805727 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.801933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:49203 (local address 127.3.214.194:44407)
0924 16:06:05.802231 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:05.802238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:05.803129 (+ 891us) server_negotiation.cc:407] Connection header received
0924 16:06:05.803324 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:05.803330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:05.803404 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:05.803501 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:05.804977 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:05.804983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:05.804987 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:05.805007 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:05.805051 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:05.805064 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:05.805186 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:05.805289 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 16:06:05.805373 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 16:06:05.806051 9186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.801946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:44407 (local address 127.3.214.193:49203)
0924 16:06:05.802934 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:05.802968 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:05.803145 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:05.803896 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:05.803899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:05.803926 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:05.804726 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:06:05.804729 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:05.804746 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:06:05.804793 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:05.805159 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:05.805164 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:05.805170 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 16:06:05.805403 (+ 233us) client_negotiation.cc:240] Negotiation successful
0924 16:06:05.805465 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":517,"threads_started":1}
I20260924 16:06:05.806201 9187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.802236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:42671 (local address 127.3.214.193:60137)
0924 16:06:05.803183 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:05.803204 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:05.803386 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:05.803751 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:05.803755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:05.803777 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:05.804624 (+ 847us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:06:05.804627 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:05.804646 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:06:05.804720 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:05.805160 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:05.805165 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:05.805170 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 16:06:05.805334 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 16:06:05.805410 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":550,"threads_started":1}
I20260924 16:06:05.807358 9113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d1458c0547468cbe8b89750ca59506" candidate_uuid: "94ca0b0c45ce4b11a232ca27399e4901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da58c42b1be346a69ca0b7a65e03def6" is_pre_election: true
I20260924 16:06:05.807415 8979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d1458c0547468cbe8b89750ca59506" candidate_uuid: "94ca0b0c45ce4b11a232ca27399e4901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab721d236394e539176764570e64ed7" is_pre_election: true
I20260924 16:06:05.807932 9113 raft_consensus.cc:2454] T 10d1458c0547468cbe8b89750ca59506 P da58c42b1be346a69ca0b7a65e03def6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ca0b0c45ce4b11a232ca27399e4901 in term 0.
I20260924 16:06:05.808081 8979 raft_consensus.cc:2454] T 10d1458c0547468cbe8b89750ca59506 P cab721d236394e539176764570e64ed7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ca0b0c45ce4b11a232ca27399e4901 in term 0.
I20260924 16:06:05.808722 8780 leader_election.cc:302] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ca0b0c45ce4b11a232ca27399e4901, da58c42b1be346a69ca0b7a65e03def6; no voters:
I20260924 16:06:05.809249 9185 raft_consensus.cc:2781] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:06:05.809427 9185 raft_consensus.cc:482] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:06:05.809542 9185 raft_consensus.cc:3037] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:05.811978 9185 raft_consensus.cc:504] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab721d236394e539176764570e64ed7" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 44407 } } peers { permanent_uuid: "da58c42b1be346a69ca0b7a65e03def6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42671 } } peers { permanent_uuid: "94ca0b0c45ce4b11a232ca27399e4901" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37955 } }
I20260924 16:06:05.812644 9185 leader_election.cc:288] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901 [CANDIDATE]: Term 1 election: Requested vote from peers cab721d236394e539176764570e64ed7 (127.3.214.194:44407), da58c42b1be346a69ca0b7a65e03def6 (127.3.214.195:42671)
I20260924 16:06:05.813266 9113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d1458c0547468cbe8b89750ca59506" candidate_uuid: "94ca0b0c45ce4b11a232ca27399e4901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da58c42b1be346a69ca0b7a65e03def6"
I20260924 16:06:05.813477 9113 raft_consensus.cc:3037] T 10d1458c0547468cbe8b89750ca59506 P da58c42b1be346a69ca0b7a65e03def6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:05.813414 8979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d1458c0547468cbe8b89750ca59506" candidate_uuid: "94ca0b0c45ce4b11a232ca27399e4901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab721d236394e539176764570e64ed7"
I20260924 16:06:05.813709 8979 raft_consensus.cc:3037] T 10d1458c0547468cbe8b89750ca59506 P cab721d236394e539176764570e64ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:05.815974 9113 raft_consensus.cc:2454] T 10d1458c0547468cbe8b89750ca59506 P da58c42b1be346a69ca0b7a65e03def6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ca0b0c45ce4b11a232ca27399e4901 in term 1.
I20260924 16:06:05.816637 8780 leader_election.cc:302] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ca0b0c45ce4b11a232ca27399e4901, da58c42b1be346a69ca0b7a65e03def6; no voters:
I20260924 16:06:05.816991 9185 raft_consensus.cc:2781] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:05.817016 8979 raft_consensus.cc:2454] T 10d1458c0547468cbe8b89750ca59506 P cab721d236394e539176764570e64ed7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ca0b0c45ce4b11a232ca27399e4901 in term 1.
I20260924 16:06:05.817490 9185 raft_consensus.cc:686] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901 [term 1 LEADER]: Becoming Leader. State: Replica: 94ca0b0c45ce4b11a232ca27399e4901, State: Running, Role: LEADER
I20260924 16:06:05.818297 9185 consensus_queue.cc:237] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab721d236394e539176764570e64ed7" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 44407 } } peers { permanent_uuid: "da58c42b1be346a69ca0b7a65e03def6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42671 } } peers { permanent_uuid: "94ca0b0c45ce4b11a232ca27399e4901" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37955 } }
I20260924 16:06:05.823834 8701 catalog_manager.cc:5495] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94ca0b0c45ce4b11a232ca27399e4901 (127.3.214.193). New cstate: current_term: 1 leader_uuid: "94ca0b0c45ce4b11a232ca27399e4901" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab721d236394e539176764570e64ed7" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 44407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da58c42b1be346a69ca0b7a65e03def6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94ca0b0c45ce4b11a232ca27399e4901" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37955 } health_report { overall_health: HEALTHY } } }
I20260924 16:06:05.892122 9176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.884940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33162 (local address 127.3.214.193:37955)
0924 16:06:05.885142 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:05.885148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:05.885677 (+ 529us) server_negotiation.cc:407] Connection header received
0924 16:06:05.885892 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:05.885897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:05.886029 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:05.886256 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:05.891332 (+ 5076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:05.891340 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:05.891345 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:05.891374 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:05.891434 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:05.891452 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:05.891658 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:05.891782 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 16:06:05.891857 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:06:05.904047 9113 raft_consensus.cc:1261] T 10d1458c0547468cbe8b89750ca59506 P da58c42b1be346a69ca0b7a65e03def6 [term 1 FOLLOWER]: Refusing update from remote peer 94ca0b0c45ce4b11a232ca27399e4901: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:05.905125 8979 raft_consensus.cc:1261] T 10d1458c0547468cbe8b89750ca59506 P cab721d236394e539176764570e64ed7 [term 1 FOLLOWER]: Refusing update from remote peer 94ca0b0c45ce4b11a232ca27399e4901: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:05.905606 9188 consensus_queue.cc:1035] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da58c42b1be346a69ca0b7a65e03def6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:05.906214 9185 consensus_queue.cc:1035] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab721d236394e539176764570e64ed7" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:05.920922 9192 mvcc.cc:205] Tried to move back new op lower bound from 7332929396326141952 to 7332929395996119040. Current Snapshot: MvccSnapshot[applied={T|T < 7332929396326141952}]
I20260924 16:06:05.940879 8702 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:05.941190 8702 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:05.945508 8702 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22c841da0d02445597414e46e71c270d: Sending DeleteTablet for 3 replicas of tablet 10d1458c0547468cbe8b89750ca59506
I20260924 16:06:05.947449 9092 tablet_service.cc:1514] Processing DeleteTablet for tablet 10d1458c0547468cbe8b89750ca59506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:05 UTC) from {username='slave'} at 127.0.0.1:55630
I20260924 16:06:05.947454 8959 tablet_service.cc:1514] Processing DeleteTablet for tablet 10d1458c0547468cbe8b89750ca59506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:05 UTC) from {username='slave'} at 127.0.0.1:34320
I20260924 16:06:05.948681 8825 tablet_service.cc:1514] Processing DeleteTablet for tablet 10d1458c0547468cbe8b89750ca59506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:05 UTC) from {username='slave'} at 127.0.0.1:33154
I20260924 16:06:05.948835 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 8762
I20260924 16:06:05.956308 9203 tablet_replica.cc:330] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901: stopping tablet replica
I20260924 16:06:05.956972 9203 raft_consensus.cc:2229] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:06:05.957819 9203 raft_consensus.cc:2258] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:05.965116 9201 tablet_replica.cc:330] T 10d1458c0547468cbe8b89750ca59506 P cab721d236394e539176764570e64ed7: stopping tablet replica
I20260924 16:06:05.965911 9201 raft_consensus.cc:2229] T 10d1458c0547468cbe8b89750ca59506 P cab721d236394e539176764570e64ed7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:06:05.966754 9201 raft_consensus.cc:2258] T 10d1458c0547468cbe8b89750ca59506 P cab721d236394e539176764570e64ed7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:05.967721 9202 tablet_replica.cc:330] T 10d1458c0547468cbe8b89750ca59506 P da58c42b1be346a69ca0b7a65e03def6: stopping tablet replica
I20260924 16:06:05.967797 9176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:05.950817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.3.214.193:37955)
0924 16:06:05.951041 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:05.951048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:05.962085 (+ 11037us) server_negotiation.cc:407] Connection header received
0924 16:06:05.963191 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:05.963200 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:05.963341 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:05.963472 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:05.966733 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:05.966742 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:05.966747 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:05.966783 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:05.966836 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:05.966857 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:05.967001 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:05.967311 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 16:06:05.967399 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 16:06:05.968217 9203 ts_tablet_manager.cc:1905] T 10d1458c0547468cbe8b89750ca59506 P 94ca0b0c45ce4b11a232ca27399e4901: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:05.969417 8886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:05.970090 9201 ts_tablet_manager.cc:1905] T 10d1458c0547468cbe8b89750ca59506 P cab721d236394e539176764570e64ed7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:05.970208 9202 raft_consensus.cc:2229] T 10d1458c0547468cbe8b89750ca59506 P da58c42b1be346a69ca0b7a65e03def6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:06:05.970943 9202 raft_consensus.cc:2258] T 10d1458c0547468cbe8b89750ca59506 P da58c42b1be346a69ca0b7a65e03def6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:05.974138 9202 ts_tablet_manager.cc:1905] T 10d1458c0547468cbe8b89750ca59506 P da58c42b1be346a69ca0b7a65e03def6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:05.978502 9201 ts_tablet_manager.cc:1918] T 10d1458c0547468cbe8b89750ca59506 P cab721d236394e539176764570e64ed7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:06:05.978928 9201 log.cc:1196] T 10d1458c0547468cbe8b89750ca59506 P cab721d236394e539176764570e64ed7: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/10d1458c0547468cbe8b89750ca59506
I20260924 16:06:05.979645 9201 ts_tablet_manager.cc:1939] T 10d1458c0547468cbe8b89750ca59506 P cab721d236394e539176764570e64ed7: Deleting consensus metadata
I20260924 16:06:05.982249 8687 catalog_manager.cc:4841] TS cab721d236394e539176764570e64ed7 (127.3.214.194:44407): tablet 10d1458c0547468cbe8b89750ca59506 (table test-table [id=2deb1de666694835ad789cbc04c99a30]) successfully deleted
I20260924 16:06:05.982323 9202 ts_tablet_manager.cc:1918] T 10d1458c0547468cbe8b89750ca59506 P da58c42b1be346a69ca0b7a65e03def6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:06:05.982514 9202 log.cc:1196] T 10d1458c0547468cbe8b89750ca59506 P da58c42b1be346a69ca0b7a65e03def6: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/10d1458c0547468cbe8b89750ca59506
I20260924 16:06:05.982991 9202 ts_tablet_manager.cc:1939] T 10d1458c0547468cbe8b89750ca59506 P da58c42b1be346a69ca0b7a65e03def6: Deleting consensus metadata
I20260924 16:06:05.984733 8689 catalog_manager.cc:4841] TS da58c42b1be346a69ca0b7a65e03def6 (127.3.214.195:42671): tablet 10d1458c0547468cbe8b89750ca59506 (table test-table [id=2deb1de666694835ad789cbc04c99a30]) successfully deleted
I20260924 16:06:06.082396 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 8762
I20260924 16:06:06.103869 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 8895
W20260924 16:06:06.103983 8687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.214.193:37955 (error 108)
I20260924 16:06:06.106537 9207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:06.105517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:37955 (local address 127.0.0.1:33178)
0924 16:06:06.106192 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:06.106357 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:37955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":369,"threads_started":1}
W20260924 16:06:06.106899 8687 catalog_manager.cc:4584] TS 94ca0b0c45ce4b11a232ca27399e4901 (127.3.214.193:37955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 10d1458c0547468cbe8b89750ca59506: Network error: Client connection negotiation failed: client connection to 127.3.214.193:37955: connect: Connection refused (error 111)
I20260924 16:06:06.107446 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:06.104822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34328 (local address 127.3.214.194:44407)
0924 16:06:06.105058 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:06.105062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:06.105158 (+ 96us) server_negotiation.cc:407] Connection header received
0924 16:06:06.105392 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:06.105397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:06.105521 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:06.105646 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:06.106724 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:06.106731 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:06.106735 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:06.106753 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:06.106789 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:06.106803 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:06.106929 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:06.107121 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 16:06:06.107188 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 16:06:06.108114 9020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:06.161142 9207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:06.160541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:37955 (local address 127.0.0.1:33184)
0924 16:06:06.160827 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:06.160965 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:37955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260924 16:06:06.162475 9207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:06.162059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:37955 (local address 127.0.0.1:33198)
0924 16:06:06.162237 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:06.162335 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:37955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 16:06:06.207417 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 8895
I20260924 16:06:06.223022 9207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:06.222573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:37955 (local address 127.0.0.1:33206)
0924 16:06:06.222769 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:06.222855 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:37955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 16:06:06.224376 9207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:06.224000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:37955 (local address 127.0.0.1:33210)
0924 16:06:06.224153 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:06.224242 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:37955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 16:06:06.227684 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 9029
I20260924 16:06:06.231101 9174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:06.228542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55640 (local address 127.3.214.195:42671)
0924 16:06:06.228805 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:06.228811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:06.228836 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:06:06.228918 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:06.228922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:06.229060 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:06.229168 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:06.230265 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:06.230271 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:06.230274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:06.230294 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:06.230338 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:06.230355 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:06.230500 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:06.230632 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 16:06:06.230697 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260924 16:06:06.231952 9153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:06.304774 9207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:06.304246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:37955 (local address 127.0.0.1:33216)
0924 16:06:06.304505 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:06.304625 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:37955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260924 16:06:06.306313 9207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:06.305434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:37955 (local address 127.0.0.1:33226)
0924 16:06:06.305829 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:06.305987 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:37955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
I20260924 16:06:06.332002 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 9029
I20260924 16:06:06.349725 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 8671
I20260924 16:06:06.351233 8733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:06.479715 9207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:06.479112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:37955 (local address 127.0.0.1:33236)
0924 16:06:06.479377 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:06.479537 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:37955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260924 16:06:06.479748 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 8671
2026-09-24T16:06:06Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3350 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 16:06:06.527894 3931 mini_oidc.cc:152] Starting JWKS server
I20260924 16:06:06.531893 3931 webserver.cc:460] Webserver started at https://127.0.0.1:38103/ using document root <none> and password file <none>
I20260924 16:06:06.532236 3931 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 16:06:06.532994 3931 webserver.cc:460] Webserver started at http://127.0.0.1:41081/ using document root <none> and password file <none>
2026-09-24T16:06:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:06Z Disabled control of system clock
I20260924 16:06:06.567987 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:39087
--webserver_interface=127.3.214.254
--webserver_port=0
--builtin_ntp_servers=127.3.214.212:46767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:39087
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38103/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265920616583-3931-0/testchainca.cert with env {}
W20260924 16:06:06.983748 9216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:06.984153 9216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:06.984284 9216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:06.992486 9216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:06.992668 9216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:06.992729 9216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:06.992816 9216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:06.992888 9216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 16:06:06.992995 9216 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 16:06:06.993052 9216 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38103/jwks.json
I20260924 16:06:07.005412 9216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:46767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:39087
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:39087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38103/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265920616583-3931-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:07.007378 9216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:07.009138 9216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:07.024009 9221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:07.024048 9224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:07.023936 9222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:07.025436 9216 server_base.cc:1034] running on GCE node
I20260924 16:06:07.026542 9216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:07.028317 9216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:07.029609 9216 hybrid_clock.cc:648] HybridClock initialized: now 1790265967029557 us; error 63 us; skew 500 ppm
I20260924 16:06:07.030042 9216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:06:07.032856 9216 webserver.cc:460] Webserver started at http://127.3.214.254:45443/ using document root <none> and password file <none>
I20260924 16:06:07.033643 9216 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:07.033743 9216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:07.034034 9216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:07.037302 9216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "3c0bf5e7838f4270bcaf44418d0f8fcb"
format_stamp: "Formatted at 2026-09-24 16:06:07 on dist-test-slave-1gc3"
server_key: "a067ff3bf3f6ffef11b67f27ac56274e"
server_key_iv: "62c963d6dea7e057d5606c59d124d9a2"
server_key_version: "encryptionkey@0"
I20260924 16:06:07.038230 9216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "3c0bf5e7838f4270bcaf44418d0f8fcb"
format_stamp: "Formatted at 2026-09-24 16:06:07 on dist-test-slave-1gc3"
server_key: "a067ff3bf3f6ffef11b67f27ac56274e"
server_key_iv: "62c963d6dea7e057d5606c59d124d9a2"
server_key_version: "encryptionkey@0"
I20260924 16:06:07.044523 9216 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 16:06:07.049145 9230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:07.051146 9216 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:06:07.051383 9216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "3c0bf5e7838f4270bcaf44418d0f8fcb"
format_stamp: "Formatted at 2026-09-24 16:06:07 on dist-test-slave-1gc3"
server_key: "a067ff3bf3f6ffef11b67f27ac56274e"
server_key_iv: "62c963d6dea7e057d5606c59d124d9a2"
server_key_version: "encryptionkey@0"
I20260924 16:06:07.051636 9216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:07.072922 9216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:07.073858 9216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:07.074226 9216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:07.095441 9216 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:39087
I20260924 16:06:07.095449 9282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:39087 every 8 connection(s)
I20260924 16:06:07.096943 9216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:07.102205 9283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:07.106747 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 9216
I20260924 16:06:07.106966 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:06:07.107604 3931 external_mini_cluster.cc:1428] Setting key 8a4dd511d9dcd5c53b9c550d867c0d64
I20260924 16:06:07.115821 9283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb: Bootstrap starting.
I20260924 16:06:07.120034 9283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:07.121560 9283 log.cc:824] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:07.124866 9286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:07.110061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37770 (local address 127.3.214.254:39087)
0924 16:06:07.111461 (+ 1400us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:07.111476 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:07.111522 (+ 46us) server_negotiation.cc:407] Connection header received
0924 16:06:07.114381 (+ 2859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:07.114422 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:07.115086 (+ 664us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:07.115560 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:07.116244 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:07.117669 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:07.119165 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:07.119941 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:07.120148 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:07.121598 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:07.121607 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:07.121627 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:07.122455 (+ 828us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:07.122593 (+ 138us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:07.122625 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:07.122770 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:07.123145 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 16:06:07.123689 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":425,"threads_started":1}
I20260924 16:06:07.125535 9283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb: No bootstrap required, opened a new log
I20260924 16:06:07.129438 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:06:07.135131 9283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c0bf5e7838f4270bcaf44418d0f8fcb" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39087 } }
I20260924 16:06:07.135679 9283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:07.135798 9283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0bf5e7838f4270bcaf44418d0f8fcb, State: Initialized, Role: FOLLOWER
I20260924 16:06:07.136510 9283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c0bf5e7838f4270bcaf44418d0f8fcb" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39087 } }
I20260924 16:06:07.136801 9283 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:07.136946 9283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:07.137135 9283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:07.139981 9283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c0bf5e7838f4270bcaf44418d0f8fcb" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39087 } }
I20260924 16:06:07.140779 9283 leader_election.cc:302] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c0bf5e7838f4270bcaf44418d0f8fcb; no voters:
I20260924 16:06:07.141481 9283 leader_election.cc:288] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:07.141675 9288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:07.142556 9288 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb [term 1 LEADER]: Becoming Leader. State: Replica: 3c0bf5e7838f4270bcaf44418d0f8fcb, State: Running, Role: LEADER
I20260924 16:06:07.143224 9288 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c0bf5e7838f4270bcaf44418d0f8fcb" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39087 } }
I20260924 16:06:07.144312 9283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:07.146827 9290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c0bf5e7838f4270bcaf44418d0f8fcb. Latest consensus state: current_term: 1 leader_uuid: "3c0bf5e7838f4270bcaf44418d0f8fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c0bf5e7838f4270bcaf44418d0f8fcb" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39087 } } }
I20260924 16:06:07.147439 9290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:07.146760 9289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c0bf5e7838f4270bcaf44418d0f8fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c0bf5e7838f4270bcaf44418d0f8fcb" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39087 } } }
I20260924 16:06:07.147915 9289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb [sys.catalog]: This master's current role is: LEADER
W20260924 16:06:07.160727 9304 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:06:07.160959 9304 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:06:07.161182 9294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:07.163327 9294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:07.172906 9294 catalog_manager.cc:1347] Generated new cluster ID: adc8645489a04337ae9c8a27cc55d3f2
I20260924 16:06:07.173054 9294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:07.191062 9294 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:07.192499 9294 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:07.208976 9294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c0bf5e7838f4270bcaf44418d0f8fcb: Generated new TSK 0
I20260924 16:06:07.210047 9294 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 16:06:07.254812 9313 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 16:06:07.255065 9313 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 16:06:07.257148 9286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:07.231830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37772 (local address 127.3.214.254:39087)
0924 16:06:07.232061 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:07.232067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:07.232668 (+ 601us) server_negotiation.cc:407] Connection header received
0924 16:06:07.233023 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:07.233028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:07.233123 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:07.233284 (+ 161us) server_negotiation.cc:226] Negotiated authn=JWT
0924 16:06:07.233919 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:07.234846 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:07.236369 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:07.236967 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:07.237107 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:07.237739 (+ 632us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 16:06:07.256373 (+ 18634us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38103/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 16:06:07.256800 (+ 427us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37772: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38103/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 16:06:07.257304 9286 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37772: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38103/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 16:06:07.258716 9312 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:07.231695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39087 (local address 127.0.0.1:37772)
0924 16:06:07.232376 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:07.232395 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:07.232760 (+ 365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:07.233323 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:07.233328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:07.233340 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0924 16:06:07.233730 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:07.233740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:07.235019 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:07.235023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:07.236198 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:07.236210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:07.237179 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:07.237187 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:07.237326 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:07.237411 (+ 85us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 16:06:07.256739 (+ 19328us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38103/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 16:06:07.257113 (+ 374us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.214.254:39087: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38103/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":332,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.214.254:39087: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38103/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 16:06:07.260939 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 9216
I20260924 16:06:07.262462 9278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:07.372061 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 9216
2026-09-24T16:06:07Z chronyd exiting
I20260924 16:06:07.423961 3931 test_util.cc:175] -----------------------------------------------
I20260924 16:06:07.424180 3931 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790265920616583-3931-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (903 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 16:06:07.428696 3931 mini_oidc.cc:152] Starting JWKS server
I20260924 16:06:07.431077 3931 webserver.cc:460] Webserver started at https://127.0.0.1:40473/ using document root <none> and password file <none>
I20260924 16:06:07.431370 3931 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 16:06:07.432209 3931 webserver.cc:460] Webserver started at http://127.0.0.1:36947/ using document root <none> and password file <none>
2026-09-24T16:06:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:07Z Disabled control of system clock
I20260924 16:06:07.469664 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:40499
--webserver_interface=127.3.214.254
--webserver_port=0
--builtin_ntp_servers=127.3.214.212:38125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:40499
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40473/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265920616583-3931-0/testchainca.cert with env {}
W20260924 16:06:07.848136 9320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:07.848522 9320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:07.848655 9320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:07.856401 9320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:07.856559 9320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:07.856616 9320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:07.856698 9320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:07.856766 9320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 16:06:07.856866 9320 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 16:06:07.856947 9320 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40473/jwks.json
I20260924 16:06:07.867465 9320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:38125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:40499
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:40499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40473/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265920616583-3931-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:07.869025 9320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:07.870450 9320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:07.883599 9325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:07.883622 9326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:07.883602 9328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:07.888223 9320 server_base.cc:1034] running on GCE node
I20260924 16:06:07.889456 9320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:07.891533 9320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:07.892814 9320 hybrid_clock.cc:648] HybridClock initialized: now 1790265967892768 us; error 43 us; skew 500 ppm
I20260924 16:06:07.893278 9320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:06:07.896481 9320 webserver.cc:460] Webserver started at http://127.3.214.254:33623/ using document root <none> and password file <none>
I20260924 16:06:07.897437 9320 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:07.897598 9320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:07.898041 9320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:07.902030 9320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "6a1316e4c2bb4c7ab63ebb30d6c42925"
format_stamp: "Formatted at 2026-09-24 16:06:07 on dist-test-slave-1gc3"
server_key: "5e03a8061f4725ca87572714f56fb37d"
server_key_iv: "23b42271519b5534ff7d076b231fd741"
server_key_version: "encryptionkey@0"
I20260924 16:06:07.903187 9320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "6a1316e4c2bb4c7ab63ebb30d6c42925"
format_stamp: "Formatted at 2026-09-24 16:06:07 on dist-test-slave-1gc3"
server_key: "5e03a8061f4725ca87572714f56fb37d"
server_key_iv: "23b42271519b5534ff7d076b231fd741"
server_key_version: "encryptionkey@0"
I20260924 16:06:07.910869 9320 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260924 16:06:07.916229 9334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:07.918375 9320 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260924 16:06:07.918725 9320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "6a1316e4c2bb4c7ab63ebb30d6c42925"
format_stamp: "Formatted at 2026-09-24 16:06:07 on dist-test-slave-1gc3"
server_key: "5e03a8061f4725ca87572714f56fb37d"
server_key_iv: "23b42271519b5534ff7d076b231fd741"
server_key_version: "encryptionkey@0"
I20260924 16:06:07.919031 9320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:07.939955 9320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:07.941136 9320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:07.941490 9320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:07.963572 9320 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:40499
I20260924 16:06:07.963618 9386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:40499 every 8 connection(s)
I20260924 16:06:07.965335 9320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:07.966795 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 9320
I20260924 16:06:07.967018 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:06:07.967736 3931 external_mini_cluster.cc:1428] Setting key 7429822c356d0fe0ad7d0d3edf459957
I20260924 16:06:07.970561 9387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:07.987229 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:07.970278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41352 (local address 127.3.214.254:40499)
0924 16:06:07.971677 (+ 1399us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:07.971701 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:07.971769 (+ 68us) server_negotiation.cc:407] Connection header received
0924 16:06:07.974451 (+ 2682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:07.974510 (+ 59us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:07.975288 (+ 778us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:07.975882 (+ 594us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:07.976706 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:07.979203 (+ 2497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:07.981050 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:07.982187 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:07.982418 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:07.984023 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:07.984036 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:07.984058 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:07.984888 (+ 830us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:07.985008 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:07.985039 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:07.985155 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:07.985439 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 16:06:07.985825 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":481,"threads_started":1}
I20260924 16:06:07.987300 9387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1316e4c2bb4c7ab63ebb30d6c42925: Bootstrap starting.
I20260924 16:06:07.990988 9387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a1316e4c2bb4c7ab63ebb30d6c42925: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:07.992198 9387 log.cc:824] T 00000000000000000000000000000000 P 6a1316e4c2bb4c7ab63ebb30d6c42925: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:07.992710 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:06:07.996008 9387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1316e4c2bb4c7ab63ebb30d6c42925: No bootstrap required, opened a new log
I20260924 16:06:08.003969 9387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a1316e4c2bb4c7ab63ebb30d6c42925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1316e4c2bb4c7ab63ebb30d6c42925" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40499 } }
I20260924 16:06:08.004496 9387 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a1316e4c2bb4c7ab63ebb30d6c42925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:08.004680 9387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a1316e4c2bb4c7ab63ebb30d6c42925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1316e4c2bb4c7ab63ebb30d6c42925, State: Initialized, Role: FOLLOWER
I20260924 16:06:08.005404 9387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a1316e4c2bb4c7ab63ebb30d6c42925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1316e4c2bb4c7ab63ebb30d6c42925" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40499 } }
I20260924 16:06:08.005698 9387 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a1316e4c2bb4c7ab63ebb30d6c42925 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:08.005856 9387 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a1316e4c2bb4c7ab63ebb30d6c42925 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:08.006040 9387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a1316e4c2bb4c7ab63ebb30d6c42925 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:08.008424 9387 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a1316e4c2bb4c7ab63ebb30d6c42925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1316e4c2bb4c7ab63ebb30d6c42925" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40499 } }
I20260924 16:06:08.009020 9387 leader_election.cc:302] T 00000000000000000000000000000000 P 6a1316e4c2bb4c7ab63ebb30d6c42925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a1316e4c2bb4c7ab63ebb30d6c42925; no voters:
I20260924 16:06:08.009773 9387 leader_election.cc:288] T 00000000000000000000000000000000 P 6a1316e4c2bb4c7ab63ebb30d6c42925 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:08.010056 9392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a1316e4c2bb4c7ab63ebb30d6c42925 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:08.011737 9392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a1316e4c2bb4c7ab63ebb30d6c42925 [term 1 LEADER]: Becoming Leader. State: Replica: 6a1316e4c2bb4c7ab63ebb30d6c42925, State: Running, Role: LEADER
I20260924 16:06:08.012199 9387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a1316e4c2bb4c7ab63ebb30d6c42925 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:08.012598 9392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a1316e4c2bb4c7ab63ebb30d6c42925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1316e4c2bb4c7ab63ebb30d6c42925" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40499 } }
I20260924 16:06:08.016626 9394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1316e4c2bb4c7ab63ebb30d6c42925 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a1316e4c2bb4c7ab63ebb30d6c42925. Latest consensus state: current_term: 1 leader_uuid: "6a1316e4c2bb4c7ab63ebb30d6c42925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1316e4c2bb4c7ab63ebb30d6c42925" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40499 } } }
I20260924 16:06:08.016649 9393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1316e4c2bb4c7ab63ebb30d6c42925 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a1316e4c2bb4c7ab63ebb30d6c42925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1316e4c2bb4c7ab63ebb30d6c42925" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40499 } } }
I20260924 16:06:08.017649 9393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1316e4c2bb4c7ab63ebb30d6c42925 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:08.017499 9394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1316e4c2bb4c7ab63ebb30d6c42925 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:08.020336 9404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:08.026187 9404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:08.037057 9404 catalog_manager.cc:1347] Generated new cluster ID: 1ef8536e3c1649f190d649a4f52b9929
I20260924 16:06:08.037232 9404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:08.056308 9404 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:08.058053 9404 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:08.067793 9404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a1316e4c2bb4c7ab63ebb30d6c42925: Generated new TSK 0
I20260924 16:06:08.069160 9404 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 16:06:08.113242 9417 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 16:06:08.113396 9417 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 16:06:08.114979 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:08.093019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41360 (local address 127.3.214.254:40499)
0924 16:06:08.093277 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:08.093285 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:08.093648 (+ 363us) server_negotiation.cc:407] Connection header received
0924 16:06:08.093966 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:08.093973 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:08.094093 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:08.094345 (+ 252us) server_negotiation.cc:226] Negotiated authn=JWT
0924 16:06:08.094890 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:08.096057 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:08.097734 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:08.098473 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:08.098626 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:08.099321 (+ 695us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 16:06:08.114012 (+ 14691us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40473/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 16:06:08.114557 (+ 545us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41360: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40473/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":109}
W20260924 16:06:08.115145 9390 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41360: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40473/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 16:06:08.115164 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:08.092751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40499 (local address 127.0.0.1:41360)
0924 16:06:08.093496 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:08.093523 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:08.093703 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:08.094370 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:08.094375 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:08.094388 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0924 16:06:08.094708 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:08.094718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:08.096268 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:08.096274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:08.097558 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:08.097574 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:08.098734 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:08.098740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:08.098904 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:08.098953 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 16:06:08.114394 (+ 15441us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40473/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 16:06:08.114739 (+ 345us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.214.254:40499: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40473/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":414,"threads_started":1}
I20260924 16:06:08.116407 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 9320
I20260924 16:06:08.118005 9382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:08.227716 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 9320
2026-09-24T16:06:08Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (858 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 16:06:08.286877 3931 mini_oidc.cc:152] Starting JWKS server
I20260924 16:06:08.289047 3931 webserver.cc:460] Webserver started at https://127.0.0.1:44853/ using document root <none> and password file <none>
I20260924 16:06:08.289302 3931 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 16:06:08.290019 3931 webserver.cc:460] Webserver started at http://127.0.0.1:41299/ using document root <none> and password file <none>
2026-09-24T16:06:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:08Z Disabled control of system clock
I20260924 16:06:08.325160 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:35393
--webserver_interface=127.3.214.254
--webserver_port=0
--builtin_ntp_servers=127.3.214.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:35393
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44853/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 16:06:08.732908 9424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:08.733363 9424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:08.733515 9424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:08.742023 9424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:08.742239 9424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:08.742313 9424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:08.742405 9424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:08.742470 9424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 16:06:08.742568 9424 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 16:06:08.742632 9424 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44853/jwks.json
I20260924 16:06:08.753769 9424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:35393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:35393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44853/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:08.755690 9424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:08.758142 9424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:08.772799 9429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:08.772846 9430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:08.772881 9432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:08.774228 9424 server_base.cc:1034] running on GCE node
I20260924 16:06:08.775300 9424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:08.777266 9424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:08.778544 9424 hybrid_clock.cc:648] HybridClock initialized: now 1790265968778473 us; error 52 us; skew 500 ppm
I20260924 16:06:08.779055 9424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:06:08.781999 9424 webserver.cc:460] Webserver started at http://127.3.214.254:40329/ using document root <none> and password file <none>
I20260924 16:06:08.782935 9424 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:08.783051 9424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:08.783439 9424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:08.787094 9424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "b773bab601da4ee1a4f670677d3998e8"
format_stamp: "Formatted at 2026-09-24 16:06:08 on dist-test-slave-1gc3"
server_key: "1b206e1f3e3f73cbecbc09071fec56c0"
server_key_iv: "dd43678ddeb60b4028fcf187a4fd3fbb"
server_key_version: "encryptionkey@0"
I20260924 16:06:08.788069 9424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "b773bab601da4ee1a4f670677d3998e8"
format_stamp: "Formatted at 2026-09-24 16:06:08 on dist-test-slave-1gc3"
server_key: "1b206e1f3e3f73cbecbc09071fec56c0"
server_key_iv: "dd43678ddeb60b4028fcf187a4fd3fbb"
server_key_version: "encryptionkey@0"
I20260924 16:06:08.794551 9424 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260924 16:06:08.799182 9438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:08.801241 9424 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:06:08.801517 9424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "b773bab601da4ee1a4f670677d3998e8"
format_stamp: "Formatted at 2026-09-24 16:06:08 on dist-test-slave-1gc3"
server_key: "1b206e1f3e3f73cbecbc09071fec56c0"
server_key_iv: "dd43678ddeb60b4028fcf187a4fd3fbb"
server_key_version: "encryptionkey@0"
I20260924 16:06:08.801823 9424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:08.825022 9424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:08.826092 9424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:08.826486 9424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:08.848528 9424 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:35393
I20260924 16:06:08.848544 9490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:35393 every 8 connection(s)
I20260924 16:06:08.850234 9424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:08.852950 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 9424
I20260924 16:06:08.853181 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:06:08.853884 3931 external_mini_cluster.cc:1428] Setting key 310a4435141559e1c696232d35c67cea
I20260924 16:06:08.854948 9491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:08.871798 9491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b773bab601da4ee1a4f670677d3998e8: Bootstrap starting.
I20260924 16:06:08.871934 9494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:08.856317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48496 (local address 127.3.214.254:35393)
0924 16:06:08.857670 (+ 1353us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:08.857687 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:08.857750 (+ 63us) server_negotiation.cc:407] Connection header received
0924 16:06:08.860258 (+ 2508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:08.860296 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:08.861087 (+ 791us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:08.861640 (+ 553us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:08.862345 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:08.864343 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:08.865925 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:08.867079 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:08.867322 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:08.868598 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:08.868611 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:08.868642 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:08.869491 (+ 849us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:08.869610 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:08.869641 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:08.869769 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:08.870026 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 16:06:08.870472 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":466,"threads_started":1}
I20260924 16:06:08.875814 9491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b773bab601da4ee1a4f670677d3998e8: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:08.877179 9491 log.cc:824] T 00000000000000000000000000000000 P b773bab601da4ee1a4f670677d3998e8: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:08.877480 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:06:08.881158 9491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b773bab601da4ee1a4f670677d3998e8: No bootstrap required, opened a new log
I20260924 16:06:08.889631 9491 raft_consensus.cc:348] T 00000000000000000000000000000000 P b773bab601da4ee1a4f670677d3998e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b773bab601da4ee1a4f670677d3998e8" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35393 } }
I20260924 16:06:08.890172 9491 raft_consensus.cc:374] T 00000000000000000000000000000000 P b773bab601da4ee1a4f670677d3998e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:08.890271 9491 raft_consensus.cc:729] T 00000000000000000000000000000000 P b773bab601da4ee1a4f670677d3998e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b773bab601da4ee1a4f670677d3998e8, State: Initialized, Role: FOLLOWER
I20260924 16:06:08.890930 9491 consensus_queue.cc:260] T 00000000000000000000000000000000 P b773bab601da4ee1a4f670677d3998e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b773bab601da4ee1a4f670677d3998e8" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35393 } }
I20260924 16:06:08.891223 9491 raft_consensus.cc:388] T 00000000000000000000000000000000 P b773bab601da4ee1a4f670677d3998e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:08.891388 9491 raft_consensus.cc:482] T 00000000000000000000000000000000 P b773bab601da4ee1a4f670677d3998e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:08.891659 9491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b773bab601da4ee1a4f670677d3998e8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:08.894325 9491 raft_consensus.cc:504] T 00000000000000000000000000000000 P b773bab601da4ee1a4f670677d3998e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b773bab601da4ee1a4f670677d3998e8" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35393 } }
I20260924 16:06:08.894965 9491 leader_election.cc:302] T 00000000000000000000000000000000 P b773bab601da4ee1a4f670677d3998e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b773bab601da4ee1a4f670677d3998e8; no voters:
I20260924 16:06:08.895663 9491 leader_election.cc:288] T 00000000000000000000000000000000 P b773bab601da4ee1a4f670677d3998e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:08.895919 9496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b773bab601da4ee1a4f670677d3998e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:08.896798 9496 raft_consensus.cc:686] T 00000000000000000000000000000000 P b773bab601da4ee1a4f670677d3998e8 [term 1 LEADER]: Becoming Leader. State: Replica: b773bab601da4ee1a4f670677d3998e8, State: Running, Role: LEADER
I20260924 16:06:08.897631 9496 consensus_queue.cc:237] T 00000000000000000000000000000000 P b773bab601da4ee1a4f670677d3998e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b773bab601da4ee1a4f670677d3998e8" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35393 } }
I20260924 16:06:08.898397 9491 sys_catalog.cc:564] T 00000000000000000000000000000000 P b773bab601da4ee1a4f670677d3998e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:08.901283 9497 sys_catalog.cc:455] T 00000000000000000000000000000000 P b773bab601da4ee1a4f670677d3998e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b773bab601da4ee1a4f670677d3998e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b773bab601da4ee1a4f670677d3998e8" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35393 } } }
I20260924 16:06:08.901360 9498 sys_catalog.cc:455] T 00000000000000000000000000000000 P b773bab601da4ee1a4f670677d3998e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b773bab601da4ee1a4f670677d3998e8. Latest consensus state: current_term: 1 leader_uuid: "b773bab601da4ee1a4f670677d3998e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b773bab601da4ee1a4f670677d3998e8" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35393 } } }
I20260924 16:06:08.901803 9497 sys_catalog.cc:458] T 00000000000000000000000000000000 P b773bab601da4ee1a4f670677d3998e8 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:08.901898 9498 sys_catalog.cc:458] T 00000000000000000000000000000000 P b773bab601da4ee1a4f670677d3998e8 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:08.905120 9502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:08.912925 9502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:08.927829 9502 catalog_manager.cc:1347] Generated new cluster ID: 42c65b8de5f340d78be90e544fe9a88f
I20260924 16:06:08.927995 9502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:08.951048 9502 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:08.952713 9502 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:08.964835 9502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b773bab601da4ee1a4f670677d3998e8: Generated new TSK 0
I20260924 16:06:08.966136 9502 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:08.996105 9494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:08.985051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.3.214.254:35393)
0924 16:06:08.985335 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:08.985342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:08.985871 (+ 529us) server_negotiation.cc:407] Connection header received
0924 16:06:08.986246 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:08.986252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:08.986377 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:08.986584 (+ 207us) server_negotiation.cc:226] Negotiated authn=JWT
0924 16:06:08.987243 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:08.988129 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:08.990450 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:08.991068 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:08.991233 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:08.992025 (+ 792us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 16:06:08.995098 (+ 3073us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44853/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 16:06:08.995703 (+ 605us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48498: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44853/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":128}
W20260924 16:06:08.996291 9494 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48498: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44853/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 16:06:08.998363 9520 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:08.984697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:35393 (local address 127.0.0.1:48498)
0924 16:06:08.985657 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:08.985687 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:08.985924 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:08.986624 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:08.986630 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:08.986646 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0924 16:06:08.987026 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:08.987040 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:08.988331 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:08.988338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:08.990231 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:08.990253 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:08.991367 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:08.991376 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:08.991594 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:08.991655 (+ 61us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 16:06:08.995569 (+ 3914us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44853/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 16:06:08.997773 (+ 2204us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.214.254:35393: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44853/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":1591,"thread_start_us":444,"threads_started":1}
I20260924 16:06:09.000437 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 9424
I20260924 16:06:09.002228 9486 generic_service.cc:196] Checking for leaks (request via RPC)
E20260924 16:06:09.008111 9521 webserver.cc:533] Webserver: sslize failed:
E20260924 16:06:09.008245 9521 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 16:06:09.103400 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 9424
2026-09-24T16:06:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (872 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T16:06:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:09Z Disabled control of system clock
I20260924 16:06:09.193593 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:32969
--webserver_interface=127.3.214.254
--webserver_port=0
--builtin_ntp_servers=127.3.214.212:39161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:32969
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265920616583-3931-0/testchainca.cert with env {}
W20260924 16:06:09.566422 9526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:09.566748 9526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:09.566836 9526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:09.574385 9526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:09.574517 9526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:09.574558 9526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:09.574633 9526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:09.574687 9526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 16:06:09.574754 9526 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 16:06:09.574796 9526 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 16:06:09.585393 9526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:39161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:32969
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:32969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265920616583-3931-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:09.586822 9526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:09.588150 9526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:09.601176 9532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:09.601243 9533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:09.601500 9535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:09.602267 9526 server_base.cc:1034] running on GCE node
I20260924 16:06:09.603420 9526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:09.605441 9526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:09.606757 9526 hybrid_clock.cc:648] HybridClock initialized: now 1790265969606697 us; error 44 us; skew 500 ppm
I20260924 16:06:09.607232 9526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:06:09.610360 9526 webserver.cc:460] Webserver started at http://127.3.214.254:44883/ using document root <none> and password file <none>
I20260924 16:06:09.611291 9526 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:09.611418 9526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:09.611835 9526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:09.615835 9526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "3014ea6d30134010bf33183541ebc42d"
format_stamp: "Formatted at 2026-09-24 16:06:09 on dist-test-slave-1gc3"
server_key: "55c067d20f19b509df4e3b910f9fa8f8"
server_key_iv: "b84a5ea64ba1d260bea9eed049ca7880"
server_key_version: "encryptionkey@0"
I20260924 16:06:09.616874 9526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "3014ea6d30134010bf33183541ebc42d"
format_stamp: "Formatted at 2026-09-24 16:06:09 on dist-test-slave-1gc3"
server_key: "55c067d20f19b509df4e3b910f9fa8f8"
server_key_iv: "b84a5ea64ba1d260bea9eed049ca7880"
server_key_version: "encryptionkey@0"
I20260924 16:06:09.623657 9526 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 16:06:09.628535 9541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:09.630647 9526 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:06:09.630939 9526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "3014ea6d30134010bf33183541ebc42d"
format_stamp: "Formatted at 2026-09-24 16:06:09 on dist-test-slave-1gc3"
server_key: "55c067d20f19b509df4e3b910f9fa8f8"
server_key_iv: "b84a5ea64ba1d260bea9eed049ca7880"
server_key_version: "encryptionkey@0"
I20260924 16:06:09.631242 9526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:09.655037 9526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:09.656229 9526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:09.656615 9526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:09.681365 9526 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:32969
I20260924 16:06:09.681382 9592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:32969 every 8 connection(s)
I20260924 16:06:09.683375 9526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:09.687903 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 9526
I20260924 16:06:09.688002 9593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:09.688130 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:06:09.688858 3931 external_mini_cluster.cc:1428] Setting key 7fea4df825339f23f56411bb25b582d2
I20260924 16:06:09.703744 9593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3014ea6d30134010bf33183541ebc42d: Bootstrap starting.
I20260924 16:06:09.707162 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:09.691483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.3.214.254:32969)
0924 16:06:09.692895 (+ 1412us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:09.692912 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:09.692970 (+ 58us) server_negotiation.cc:407] Connection header received
0924 16:06:09.695231 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:09.695265 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:09.695948 (+ 683us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:09.696420 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:09.697251 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:09.698889 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:09.700901 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:09.701834 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:09.702011 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:09.703547 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:09.703557 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:09.703581 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:06:09.704380 (+ 799us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:09.704502 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:06:09.704532 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:09.704672 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:09.705049 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 16:06:09.705632 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":837,"thread_start_us":503,"threads_started":1}
I20260924 16:06:09.707603 9593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3014ea6d30134010bf33183541ebc42d: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:09.708847 9593 log.cc:824] T 00000000000000000000000000000000 P 3014ea6d30134010bf33183541ebc42d: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:09.712839 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:06:09.713199 9593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3014ea6d30134010bf33183541ebc42d: No bootstrap required, opened a new log
I20260924 16:06:09.721899 9593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3014ea6d30134010bf33183541ebc42d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3014ea6d30134010bf33183541ebc42d" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 32969 } }
I20260924 16:06:09.722380 9593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3014ea6d30134010bf33183541ebc42d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:09.722469 9593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3014ea6d30134010bf33183541ebc42d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3014ea6d30134010bf33183541ebc42d, State: Initialized, Role: FOLLOWER
I20260924 16:06:09.723058 9593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3014ea6d30134010bf33183541ebc42d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3014ea6d30134010bf33183541ebc42d" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 32969 } }
I20260924 16:06:09.723287 9593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3014ea6d30134010bf33183541ebc42d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:09.723428 9593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3014ea6d30134010bf33183541ebc42d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:09.723587 9593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3014ea6d30134010bf33183541ebc42d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:09.726038 9593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3014ea6d30134010bf33183541ebc42d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3014ea6d30134010bf33183541ebc42d" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 32969 } }
I20260924 16:06:09.726646 9593 leader_election.cc:302] T 00000000000000000000000000000000 P 3014ea6d30134010bf33183541ebc42d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3014ea6d30134010bf33183541ebc42d; no voters:
I20260924 16:06:09.727398 9593 leader_election.cc:288] T 00000000000000000000000000000000 P 3014ea6d30134010bf33183541ebc42d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:09.727727 9598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3014ea6d30134010bf33183541ebc42d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:09.728899 9598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3014ea6d30134010bf33183541ebc42d [term 1 LEADER]: Becoming Leader. State: Replica: 3014ea6d30134010bf33183541ebc42d, State: Running, Role: LEADER
I20260924 16:06:09.729576 9598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3014ea6d30134010bf33183541ebc42d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3014ea6d30134010bf33183541ebc42d" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 32969 } }
I20260924 16:06:09.729951 9593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3014ea6d30134010bf33183541ebc42d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:09.733896 9599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3014ea6d30134010bf33183541ebc42d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3014ea6d30134010bf33183541ebc42d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3014ea6d30134010bf33183541ebc42d" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 32969 } } }
I20260924 16:06:09.733824 9600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3014ea6d30134010bf33183541ebc42d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3014ea6d30134010bf33183541ebc42d. Latest consensus state: current_term: 1 leader_uuid: "3014ea6d30134010bf33183541ebc42d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3014ea6d30134010bf33183541ebc42d" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 32969 } } }
I20260924 16:06:09.734764 9599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3014ea6d30134010bf33183541ebc42d [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:09.735090 9600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3014ea6d30134010bf33183541ebc42d [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:09.742632 9608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:09.748971 9608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:09.757882 9608 catalog_manager.cc:1347] Generated new cluster ID: 431dbe32621941d08592dbfa5e132876
I20260924 16:06:09.758040 9608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:09.786623 9608 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:09.788077 9608 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:09.799031 9608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3014ea6d30134010bf33183541ebc42d: Generated new TSK 0
I20260924 16:06:09.800583 9608 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:09.932132 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:09.822081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42424 (local address 127.3.214.254:32969)
0924 16:06:09.822375 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:09.822383 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:09.823110 (+ 727us) server_negotiation.cc:407] Connection header received
0924 16:06:09.823475 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:09.823480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:09.823573 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:09.823739 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0924 16:06:09.824430 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:09.825283 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:09.827669 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:09.828205 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:09.828345 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:09.829080 (+ 735us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 16:06:09.931444 (+102364us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 16:06:09.931871 (+ 427us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42424: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":102}
W20260924 16:06:09.932492 9596 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42424: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 16:06:09.932816 9622 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:09.821855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:32969 (local address 127.0.0.1:42424)
0924 16:06:09.822929 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:09.822955 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:09.823202 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:09.823779 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:09.823785 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:09.823800 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0924 16:06:09.824214 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:09.824229 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:09.825486 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:09.825494 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:09.827445 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:09.827465 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:09.828427 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:09.828441 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:09.828649 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:09.828715 (+ 66us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 16:06:09.931894 (+103179us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 16:06:09.932308 (+ 414us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.214.254:32969: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":887,"thread_start_us":660,"threads_started":1}
I20260924 16:06:09.934410 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 9526
I20260924 16:06:09.935717 9588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:10.053151 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 9526
2026-09-24T16:06:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (931 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (462 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (451 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:11 dist-test-slave-1gc3 krb5kdc[9637](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:11 dist-test-slave-1gc3 krb5kdc[9637](info): set up 2 sockets
Sep 24 16:06:11 dist-test-slave-1gc3 krb5kdc[9637](info): commencing operation
krb5kdc: starting...
W20260924 16:06:12.008905 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.006s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:06:12 dist-test-slave-1gc3 krb5kdc[9637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:06:12.258275 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:39091
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:35227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:39091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:12.610893 9650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:12.611234 9650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:12.611351 9650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:12.619048 9650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:12.619179 9650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:12.619235 9650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:12.619328 9650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:12.619388 9650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:12.629741 9650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:35227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:39091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:39091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:12.631227 9650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:12.632633 9650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:12.645982 9655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:12.646042 9656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:12.646068 9659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:12.647251 9650 server_base.cc:1034] running on GCE node
I20260924 16:06:12.648303 9650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:12.650225 9650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:12.651441 9650 hybrid_clock.cc:648] HybridClock initialized: now 1790265972651368 us; error 66 us; skew 500 ppm
I20260924 16:06:12.651865 9650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:12 dist-test-slave-1gc3 krb5kdc[9637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265972, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:12.655762 9650 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:06:12.657644 9650 webserver.cc:460] Webserver started at http://127.3.214.254:38261/ using document root <none> and password file <none>
I20260924 16:06:12.658550 9650 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:12.658681 9650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:12.659008 9650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:12.662657 9650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "99901c23c7a940a09c2d4e54ef6d4020"
format_stamp: "Formatted at 2026-09-24 16:06:12 on dist-test-slave-1gc3"
server_key: "c0953e87a14f321fdfc75bebf8a2b5e4"
server_key_iv: "7dc23bd9bad4ecb741bf282aa6b6e140"
server_key_version: "encryptionkey@0"
I20260924 16:06:12.663611 9650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "99901c23c7a940a09c2d4e54ef6d4020"
format_stamp: "Formatted at 2026-09-24 16:06:12 on dist-test-slave-1gc3"
server_key: "c0953e87a14f321fdfc75bebf8a2b5e4"
server_key_iv: "7dc23bd9bad4ecb741bf282aa6b6e140"
server_key_version: "encryptionkey@0"
I20260924 16:06:12.670377 9650 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 16:06:12.675007 9666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:12.676841 9650 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:06:12.677075 9650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "99901c23c7a940a09c2d4e54ef6d4020"
format_stamp: "Formatted at 2026-09-24 16:06:12 on dist-test-slave-1gc3"
server_key: "c0953e87a14f321fdfc75bebf8a2b5e4"
server_key_iv: "7dc23bd9bad4ecb741bf282aa6b6e140"
server_key_version: "encryptionkey@0"
I20260924 16:06:12.677342 9650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:12.703915 9650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:12.709066 9650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:12.709438 9650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:12.731098 9650 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:39091
I20260924 16:06:12.731153 9718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:39091 every 8 connection(s)
I20260924 16:06:12.732892 9650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:12.735532 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 9650
I20260924 16:06:12.735774 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:06:12.736613 3931 external_mini_cluster.cc:1428] Setting key eabf14ad8b651835f5ed71c1d2889fce
I20260924 16:06:12.738049 9719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 16:06:12 dist-test-slave-1gc3 krb5kdc[9637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:06:12.754644 9719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020: Bootstrap starting.
I20260924 16:06:12.759140 9719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:12.760403 9719 log.cc:824] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:12.763757 9719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020: No bootstrap required, opened a new log
I20260924 16:06:12.764719 9722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:12.739173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38804 (local address 127.3.214.254:39091)
0924 16:06:12.740132 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:12.740148 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:12.740203 (+ 55us) server_negotiation.cc:407] Connection header received
0924 16:06:12.742085 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:12.742146 (+ 61us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:12.742703 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:12.743150 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:12.745040 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:12.746723 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:12.748499 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:12.749273 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:12.749441 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:12.755169 (+ 5728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:12.755184 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:12.755210 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:12.755279 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:12.759350 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:12.760263 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:12.760273 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:12.760282 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:12.760541 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:12.761003 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:12.761008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:12.761010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:12.761827 (+ 817us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:12.762233 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:12.762710 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 16:06:12.763255 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":419,"threads_started":1}
I20260924 16:06:12.772261 9719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99901c23c7a940a09c2d4e54ef6d4020" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39091 } }
I20260924 16:06:12.772725 9719 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:12.772881 9719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99901c23c7a940a09c2d4e54ef6d4020, State: Initialized, Role: FOLLOWER
I20260924 16:06:12.773530 9719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99901c23c7a940a09c2d4e54ef6d4020" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39091 } }
I20260924 16:06:12.773748 9719 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:12.773859 9719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:12.773986 9719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:12.776238 9719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99901c23c7a940a09c2d4e54ef6d4020" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39091 } }
I20260924 16:06:12.776728 9719 leader_election.cc:302] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99901c23c7a940a09c2d4e54ef6d4020; no voters:
I20260924 16:06:12.777343 9719 leader_election.cc:288] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:12.777566 9724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:12.778674 9724 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020 [term 1 LEADER]: Becoming Leader. State: Replica: 99901c23c7a940a09c2d4e54ef6d4020, State: Running, Role: LEADER
I20260924 16:06:12.779340 9724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99901c23c7a940a09c2d4e54ef6d4020" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39091 } }
I20260924 16:06:12.779767 9719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:12.783118 9726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99901c23c7a940a09c2d4e54ef6d4020. Latest consensus state: current_term: 1 leader_uuid: "99901c23c7a940a09c2d4e54ef6d4020" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99901c23c7a940a09c2d4e54ef6d4020" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39091 } } }
I20260924 16:06:12.783593 9726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:12.784211 9725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99901c23c7a940a09c2d4e54ef6d4020" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99901c23c7a940a09c2d4e54ef6d4020" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39091 } } }
I20260924 16:06:12.784565 9725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020 [sys.catalog]: This master's current role is: LEADER
W20260924 16:06:12.793773 9742 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:06:12.793958 9742 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:06:12.794463 9735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:12.796618 9735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:12.807597 9735 catalog_manager.cc:1347] Generated new cluster ID: 400b354d9b544ffcb7279655ec1c61c0
I20260924 16:06:12.807773 9735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:12.831425 9735 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:12.833282 9735 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:12.844564 9735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99901c23c7a940a09c2d4e54ef6d4020: Generated new TSK 0
I20260924 16:06:12.845933 9735 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.214.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
I20260924 16:06:12.917707 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39091
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:35227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:13.273886 9747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:13.274230 9747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:13.274333 9747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:13.281875 9747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:13.282019 9747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:13.282202 9747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:06:13.292725 9747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:35227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:13.294167 9747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:13.295547 9747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:13.310233 9755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:13.310284 9753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:13.310559 9752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:13.311250 9747 server_base.cc:1034] running on GCE node
I20260924 16:06:13.312000 9747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:13.313135 9747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:13.314433 9747 hybrid_clock.cc:648] HybridClock initialized: now 1790265973314347 us; error 88 us; skew 500 ppm
I20260924 16:06:13.314810 9747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:13 dist-test-slave-1gc3 krb5kdc[9637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265973, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:13.318624 9747 init.cc:377] Logged in from keytab as kudu/127.3.214.193@KRBTEST.COM (short username kudu)
I20260924 16:06:13.320456 9747 webserver.cc:460] Webserver started at http://127.3.214.193:36573/ using document root <none> and password file <none>
I20260924 16:06:13.321230 9747 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:13.321369 9747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:13.321698 9747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:13.325016 9747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "b61273601f8a4f0ebfba308b480a5cc2"
format_stamp: "Formatted at 2026-09-24 16:06:13 on dist-test-slave-1gc3"
server_key: "8a79ca668bc9429c8dbe3f6e17ae2ced"
server_key_iv: "d976c69bdd01478b1173d9c11c324aa7"
server_key_version: "encryptionkey@0"
I20260924 16:06:13.325999 9747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "b61273601f8a4f0ebfba308b480a5cc2"
format_stamp: "Formatted at 2026-09-24 16:06:13 on dist-test-slave-1gc3"
server_key: "8a79ca668bc9429c8dbe3f6e17ae2ced"
server_key_iv: "d976c69bdd01478b1173d9c11c324aa7"
server_key_version: "encryptionkey@0"
I20260924 16:06:13.332435 9747 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 16:06:13.337049 9762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:13.339066 9747 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:06:13.339282 9747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "b61273601f8a4f0ebfba308b480a5cc2"
format_stamp: "Formatted at 2026-09-24 16:06:13 on dist-test-slave-1gc3"
server_key: "8a79ca668bc9429c8dbe3f6e17ae2ced"
server_key_iv: "d976c69bdd01478b1173d9c11c324aa7"
server_key_version: "encryptionkey@0"
I20260924 16:06:13.339516 9747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:13.366073 9747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:13.371249 9747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:13.371558 9747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:13.372668 9747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:13.374246 9747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:13.374326 9747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:13.374459 9747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:13.374511 9747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:13.410867 9747 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:44765
I20260924 16:06:13.410972 9876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:44765 every 8 connection(s)
I20260924 16:06:13.412261 9747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:06:13.416117 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 9747
I20260924 16:06:13.416337 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:06:13.417176 3931 external_mini_cluster.cc:1428] Setting key a053e04ca1e368b6a79415443d8406c7
Sep 24 16:06:13 dist-test-slave-1gc3 krb5kdc[9637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265973, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:06:13.437518 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:13.415746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:46551 (local address 127.3.214.254:39091)
0924 16:06:13.416297 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:13.416304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:13.417691 (+ 1387us) server_negotiation.cc:407] Connection header received
0924 16:06:13.420406 (+ 2715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:13.420411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:13.420511 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:13.420612 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:13.422669 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:13.423473 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:13.425267 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:13.425726 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:13.425855 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:13.429942 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:13.429948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:13.429952 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:13.429994 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:13.433111 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:13.434314 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:13.434323 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:13.434327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:13.434608 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:13.435159 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:13.435164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:13.435168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:13.435601 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:13.435796 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:13.436800 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0924 16:06:13.437114 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":282,"threads_started":1}
WARNING: no policy specified for kudu/127.3.214.194@KRBTEST.COM; defaulting to no policy
I20260924 16:06:13.438815 9880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:13.416395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39091 (local address 127.3.214.193:46551)
0924 16:06:13.417450 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:13.417512 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:13.420070 (+ 2558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:13.420931 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:13.420941 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:13.421619 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:13.422464 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:13.422489 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:13.423640 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:13.423645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:13.425087 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:13.425108 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:13.425999 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:13.426007 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:13.426339 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:13.427106 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:13.427141 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:13.429657 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:13.433348 (+ 3691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:13.433360 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:13.433382 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:13.434004 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:13.434763 (+ 759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:13.434766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:13.434769 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:13.435013 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:13.435773 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:13.435781 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:13.436301 (+ 520us) client_negotiation.cc:769] Sending connection context
0924 16:06:13.436609 (+ 308us) client_negotiation.cc:240] Negotiation successful
0924 16:06:13.437219 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":415,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.194@KRBTEST.COM" created.
I20260924 16:06:13.440958 9877 heartbeater.cc:347] Connected to a master server at 127.3.214.254:39091
I20260924 16:06:13.441458 9877 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:13.442579 9877 heartbeater.cc:511] Master 127.3.214.254:39091 requested a full tablet report, sending...
I20260924 16:06:13.446053 9683 ts_manager.cc:194] Registered new tserver with Master: b61273601f8a4f0ebfba308b480a5cc2 (127.3.214.193:44765)
I20260924 16:06:13.448396 9683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:46551
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
I20260924 16:06:13.471817 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39091
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:35227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:13.878319 9883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:13.878736 9883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:13.878895 9883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:13.887447 9883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:13.887640 9883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:13.887835 9883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:06:13.899544 9883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:35227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:13.901394 9883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:13.903183 9883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:13.921471 9891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:13.921556 9889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:13.921800 9888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:13.922803 9883 server_base.cc:1034] running on GCE node
I20260924 16:06:13.923656 9883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:13.924976 9883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:13.926245 9883 hybrid_clock.cc:648] HybridClock initialized: now 1790265973926170 us; error 53 us; skew 500 ppm
I20260924 16:06:13.926646 9883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:13 dist-test-slave-1gc3 krb5kdc[9637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265973, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:13.930439 9883 init.cc:377] Logged in from keytab as kudu/127.3.214.194@KRBTEST.COM (short username kudu)
I20260924 16:06:13.932287 9883 webserver.cc:460] Webserver started at http://127.3.214.194:37791/ using document root <none> and password file <none>
I20260924 16:06:13.933176 9883 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:13.933322 9883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:13.933763 9883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:13.938577 9883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "b4edf5ec294d4a9c9f236e956da1fe84"
format_stamp: "Formatted at 2026-09-24 16:06:13 on dist-test-slave-1gc3"
server_key: "b5bd4b530a24357dad9f47dd77068fa6"
server_key_iv: "cb315cd29708922ebecaf17dd056cc91"
server_key_version: "encryptionkey@0"
I20260924 16:06:13.940237 9883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "b4edf5ec294d4a9c9f236e956da1fe84"
format_stamp: "Formatted at 2026-09-24 16:06:13 on dist-test-slave-1gc3"
server_key: "b5bd4b530a24357dad9f47dd77068fa6"
server_key_iv: "cb315cd29708922ebecaf17dd056cc91"
server_key_version: "encryptionkey@0"
I20260924 16:06:13.949035 9883 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260924 16:06:13.953912 9899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:13.955892 9883 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:06:13.956120 9883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "b4edf5ec294d4a9c9f236e956da1fe84"
format_stamp: "Formatted at 2026-09-24 16:06:13 on dist-test-slave-1gc3"
server_key: "b5bd4b530a24357dad9f47dd77068fa6"
server_key_iv: "cb315cd29708922ebecaf17dd056cc91"
server_key_version: "encryptionkey@0"
I20260924 16:06:13.956400 9883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:13.987572 9883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:13.993510 9883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:13.993930 9883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:13.995424 9883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:13.997355 9883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:13.997474 9883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:13.997637 9883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:13.997704 9883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:14.037178 9883 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:39975
I20260924 16:06:14.037271 10012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:39975 every 8 connection(s)
I20260924 16:06:14.038800 9883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:06:14.040817 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 9883
I20260924 16:06:14.041018 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:06:14.041587 3931 external_mini_cluster.cc:1428] Setting key 9f976179200e1f5787b56df75d2ca58c
Sep 24 16:06:14 dist-test-slave-1gc3 krb5kdc[9637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265973, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:06:14.066438 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:14.042470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:33085 (local address 127.3.214.254:39091)
0924 16:06:14.043009 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:14.043015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:14.044681 (+ 1666us) server_negotiation.cc:407] Connection header received
0924 16:06:14.047637 (+ 2956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:14.047642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:14.047734 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:14.047875 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:14.050256 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:14.051196 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:14.053642 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:14.054411 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:14.054718 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:14.058937 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:14.058945 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:14.058950 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:14.059009 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:14.062619 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:14.063663 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:14.063669 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:14.063672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:14.063835 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:14.064263 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:14.064267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:14.064270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:14.064659 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:14.064838 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:14.065667 (+ 829us) server_negotiation.cc:299] Negotiation successful
0924 16:06:14.065937 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":271,"threads_started":1}
I20260924 16:06:14.067377 10016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:14.043249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39091 (local address 127.3.214.194:33085)
0924 16:06:14.044459 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:14.044529 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:14.047197 (+ 2668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:14.048207 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:14.048221 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:14.049027 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:14.049942 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:14.049971 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:14.051453 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:14.051459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:14.053357 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:14.053378 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:14.054739 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:14.054746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:14.054998 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:14.055748 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:14.055776 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:14.058539 (+ 2763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:14.062873 (+ 4334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:14.062882 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:14.062899 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:14.063502 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:14.063936 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:14.063939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:14.063941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:14.064153 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:14.064824 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:14.064831 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:14.065251 (+ 420us) client_negotiation.cc:769] Sending connection context
0924 16:06:14.065524 (+ 273us) client_negotiation.cc:240] Negotiation successful
0924 16:06:14.066008 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":430,"threads_started":1}
I20260924 16:06:14.069020 10013 heartbeater.cc:347] Connected to a master server at 127.3.214.254:39091
I20260924 16:06:14.069343 10013 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.214.195@KRBTEST.COM; defaulting to no policy
I20260924 16:06:14.070154 10013 heartbeater.cc:511] Master 127.3.214.254:39091 requested a full tablet report, sending...
I20260924 16:06:14.071872 9683 ts_manager.cc:194] Registered new tserver with Master: b4edf5ec294d4a9c9f236e956da1fe84 (127.3.214.194:39975)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.195@KRBTEST.COM" created.
I20260924 16:06:14.072858 9683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:33085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
I20260924 16:06:14.102954 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39091
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:35227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:06:14.452696 9877 heartbeater.cc:503] Master 127.3.214.254:39091 was elected leader, sending a full tablet report...
W20260924 16:06:14.467691 10019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:14.467986 10019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:14.468096 10019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:14.475579 10019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:14.475703 10019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:14.475849 10019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:06:14.486172 10019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:35227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:14.487633 10019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:14.489012 10019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:14.503466 10024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:14.503425 10027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:14.503769 10025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:14.505123 10019 server_base.cc:1034] running on GCE node
I20260924 16:06:14.505941 10019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:14.507102 10019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:14.508461 10019 hybrid_clock.cc:648] HybridClock initialized: now 1790265974508346 us; error 86 us; skew 500 ppm
I20260924 16:06:14.508960 10019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:14 dist-test-slave-1gc3 krb5kdc[9637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265974, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:14.512445 10019 init.cc:377] Logged in from keytab as kudu/127.3.214.195@KRBTEST.COM (short username kudu)
I20260924 16:06:14.514241 10019 webserver.cc:460] Webserver started at http://127.3.214.195:35167/ using document root <none> and password file <none>
I20260924 16:06:14.514981 10019 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:14.515105 10019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:14.515422 10019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:14.518709 10019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "c39a62776e83457299ec38d2045aa580"
format_stamp: "Formatted at 2026-09-24 16:06:14 on dist-test-slave-1gc3"
server_key: "f4e1e2e50f34cc9c2c0725357b973680"
server_key_iv: "5fb68d6de93207219d813731fbf47626"
server_key_version: "encryptionkey@0"
I20260924 16:06:14.519629 10019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "c39a62776e83457299ec38d2045aa580"
format_stamp: "Formatted at 2026-09-24 16:06:14 on dist-test-slave-1gc3"
server_key: "f4e1e2e50f34cc9c2c0725357b973680"
server_key_iv: "5fb68d6de93207219d813731fbf47626"
server_key_version: "encryptionkey@0"
I20260924 16:06:14.525960 10019 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 16:06:14.530141 10034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:14.532364 10019 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:06:14.532675 10019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "c39a62776e83457299ec38d2045aa580"
format_stamp: "Formatted at 2026-09-24 16:06:14 on dist-test-slave-1gc3"
server_key: "f4e1e2e50f34cc9c2c0725357b973680"
server_key_iv: "5fb68d6de93207219d813731fbf47626"
server_key_version: "encryptionkey@0"
I20260924 16:06:14.533020 10019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:14.554057 10019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:14.558962 10019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:14.559263 10019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:14.560499 10019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:14.562288 10019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:14.562372 10019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:14.562481 10019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:14.562528 10019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:14.608471 10019 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:32899
I20260924 16:06:14.608533 10148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:32899 every 8 connection(s)
I20260924 16:06:14.609941 10019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:06:14.620234 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 10019
I20260924 16:06:14.620404 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:06:14.620874 3931 external_mini_cluster.cc:1428] Setting key decbc8cf251ee6b6062d0f1f51bd1caa
Sep 24 16:06:14 dist-test-slave-1gc3 krb5kdc[9637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265974, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:06:14.635833 10151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:14.613239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:43547 (local address 127.3.214.254:39091)
0924 16:06:14.613771 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:14.613781 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:14.615353 (+ 1572us) server_negotiation.cc:407] Connection header received
0924 16:06:14.617705 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:14.617712 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:14.617821 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:14.617937 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:14.619988 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:14.620985 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:14.622832 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:14.623331 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:14.623463 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:14.628156 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:14.628162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:14.628166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:14.628215 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:14.631447 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:14.632669 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:14.632674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:14.632677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:14.632878 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:14.633474 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:14.633481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:14.633484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:14.633883 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:14.634040 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:14.635049 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0924 16:06:14.635381 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":261,"threads_started":1}
I20260924 16:06:14.637006 10152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:14.613854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39091 (local address 127.3.214.195:43547)
0924 16:06:14.615101 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:14.615160 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:14.617351 (+ 2191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:14.618334 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:14.618345 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:14.619009 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:14.619779 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:14.619801 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:14.621159 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:14.621164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:14.622667 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:14.622682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:14.623627 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:14.623636 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:14.623913 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:14.624733 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:14.624763 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:14.627756 (+ 2993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:14.631751 (+ 3995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:14.631764 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:14.631782 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:14.632434 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:14.633055 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:14.633059 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:14.633062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:14.633319 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:14.634004 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:14.634011 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:14.634531 (+ 520us) client_negotiation.cc:769] Sending connection context
0924 16:06:14.634879 (+ 348us) client_negotiation.cc:240] Negotiation successful
0924 16:06:14.635454 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"thread_start_us":514,"threads_started":1}
I20260924 16:06:14.638743 10149 heartbeater.cc:347] Connected to a master server at 127.3.214.254:39091
I20260924 16:06:14.639228 10149 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:14.640376 10149 heartbeater.cc:511] Master 127.3.214.254:39091 requested a full tablet report, sending...
I20260924 16:06:14.642544 9683 ts_manager.cc:194] Registered new tserver with Master: c39a62776e83457299ec38d2045aa580 (127.3.214.195:32899)
I20260924 16:06:14.643745 9683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:43547
I20260924 16:06:14.650655 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/test-admin.keytab.
Sep 24 16:06:14 dist-test-slave-1gc3 krb5kdc[9637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:14.681100 3931 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 16:06:14.687330 10157 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:14.684579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51178 (local address 127.3.214.193:44765)
0924 16:06:14.685354 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:14.685363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:14.685397 (+ 34us) server_negotiation.cc:407] Connection header received
0924 16:06:14.685472 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:14.685478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:14.685714 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:14.685869 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:14.686925 (+ 1056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51178: BlockingRecv error: recv got EOF from 127.0.0.1:51178 (error 108)
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":363,"threads_started":1}
W20260924 16:06:14.687892 3931 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/krb5cc' of type FILE
Sep 24 16:06:14 dist-test-slave-1gc3 krb5kdc[9637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:14.689599 3931 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 16:06:14 dist-test-slave-1gc3 krb5kdc[9637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:14.691165 3931 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 16:06:14 dist-test-slave-1gc3 krb5kdc[9637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.193@KRBTEST.COM
I20260924 16:06:14.708961 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:14.693056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51186 (local address 127.3.214.193:44765)
0924 16:06:14.693345 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:14.693352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:14.693601 (+ 249us) server_negotiation.cc:407] Connection header received
0924 16:06:14.693801 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:14.693806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:14.693877 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:14.693980 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:14.695098 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:14.696150 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:14.697602 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:14.698311 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:14.698446 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:14.702447 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:14.702463 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:14.702481 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:14.702536 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:14.705574 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:14.706544 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:14.706554 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:14.706558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:14.706742 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:14.707168 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:14.707171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:14.707173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:14.707599 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:14.707841 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:14.708301 (+ 460us) server_negotiation.cc:299] Negotiation successful
0924 16:06:14.708615 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
W20260924 16:06:14.716800 10157 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:14.714541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51194 (local address 127.3.214.193:44765)
0924 16:06:14.714826 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:14.714833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:14.715104 (+ 271us) server_negotiation.cc:407] Connection header received
0924 16:06:14.715324 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:14.715328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:14.715402 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:14.715493 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:14.716488 (+ 995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51194: BlockingRecv error: recv got EOF from 127.0.0.1:51194 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 16:06:14.717067 3931 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/krb5cc' of type FILE
Sep 24 16:06:14 dist-test-slave-1gc3 krb5kdc[9637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:14.718915 3931 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 16:06:14 dist-test-slave-1gc3 krb5kdc[9637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:14.720414 3931 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 16:06:14 dist-test-slave-1gc3 krb5kdc[9637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.193@KRBTEST.COM
I20260924 16:06:14.739530 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:14.722336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51202 (local address 127.3.214.193:44765)
0924 16:06:14.722599 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:14.722607 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:14.722997 (+ 390us) server_negotiation.cc:407] Connection header received
0924 16:06:14.723217 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:14.723224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:14.723329 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:14.723449 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:14.724494 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:14.725314 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:14.726686 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:14.727183 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:14.727299 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:14.731203 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:14.731210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:14.731214 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:14.731265 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:14.735508 (+ 4243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:14.736517 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:14.736524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:14.736528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:14.736719 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:14.737228 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:14.737233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:14.737237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:14.737655 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:14.737964 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:14.738804 (+ 840us) server_negotiation.cc:299] Negotiation successful
0924 16:06:14.739118 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 16:06:14.745826 10157 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:14.743596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51216 (local address 127.3.214.193:44765)
0924 16:06:14.743891 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:14.743899 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:14.744266 (+ 367us) server_negotiation.cc:407] Connection header received
0924 16:06:14.744468 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:14.744472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:14.744559 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:14.744649 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:14.745549 (+ 900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51216: BlockingRecv error: recv got EOF from 127.0.0.1:51216 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260924 16:06:14.746059 3931 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790265920616583-3931-0/krb5kdc/krb5cc' of type FILE
Sep 24 16:06:14 dist-test-slave-1gc3 krb5kdc[9637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:14.747862 3931 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 16:06:14 dist-test-slave-1gc3 krb5kdc[9637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:14.749349 3931 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 16:06:14 dist-test-slave-1gc3 krb5kdc[9637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.193@KRBTEST.COM
I20260924 16:06:14.769431 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:14.751024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51228 (local address 127.3.214.193:44765)
0924 16:06:14.751277 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:14.751285 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:14.751531 (+ 246us) server_negotiation.cc:407] Connection header received
0924 16:06:14.751711 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:14.751717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:14.751788 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:14.751875 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:14.752885 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:14.753644 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:14.755370 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:14.756088 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:14.756268 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:14.761224 (+ 4956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:14.761230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:14.761235 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:14.761290 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:14.764927 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:14.765946 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:14.765951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:14.765955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:14.766207 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:14.766676 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:14.766680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:14.766683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:14.767088 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:14.767408 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:14.768689 (+ 1281us) server_negotiation.cc:299] Negotiation successful
0924 16:06:14.768984 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 16:06:14.771601 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 9747
I20260924 16:06:14.785735 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:14.772595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51244 (local address 127.3.214.193:44765)
0924 16:06:14.772846 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:14.772853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:14.773214 (+ 361us) server_negotiation.cc:407] Connection header received
0924 16:06:14.773380 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:14.773385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:14.773469 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:14.773583 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:14.774824 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:14.775575 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:14.776834 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:14.777314 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:14.777418 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:14.779703 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:14.779708 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:14.779711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:14.779748 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:14.782715 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:14.783627 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:14.783632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:14.783635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:14.783803 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:14.784213 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:14.784216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:14.784218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:14.784535 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:14.784795 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:14.785088 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 16:06:14.785297 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260924 16:06:14.786693 9872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:14.875572 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 9747
I20260924 16:06:14.893028 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 9883
Sep 24 16:06:14 dist-test-slave-1gc3 krb5kdc[9637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.194@KRBTEST.COM
I20260924 16:06:14.910339 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:14.894068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45006 (local address 127.3.214.194:39975)
0924 16:06:14.894646 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:14.894654 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:14.894686 (+ 32us) server_negotiation.cc:407] Connection header received
0924 16:06:14.894758 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:14.894766 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:14.895019 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:14.895191 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:14.896476 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:14.897522 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:14.899034 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:14.899613 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:14.899737 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:14.904055 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:14.904074 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:14.904095 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:14.904148 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:14.907291 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:14.908118 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:14.908126 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:14.908129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:14.908289 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:14.908666 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:14.908670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:14.908672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:14.909059 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:14.909316 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:14.909699 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 16:06:14.909957 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":255,"threads_started":1}
I20260924 16:06:14.912086 10008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:15.003693 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 9883
I20260924 16:06:15.022948 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 10019
Sep 24 16:06:15 dist-test-slave-1gc3 krb5kdc[9637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.195@KRBTEST.COM
I20260924 16:06:15.041728 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:15.024326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39324 (local address 127.3.214.195:32899)
0924 16:06:15.025135 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:15.025143 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:15.025179 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:06:15.025253 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:15.025261 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:15.025493 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:15.025644 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:15.027057 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:15.028179 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:15.029633 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:15.030335 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:15.030470 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:15.035264 (+ 4794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:15.035281 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:15.035302 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:15.035361 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:15.038459 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:15.039414 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:15.039422 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:15.039425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:15.039601 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:15.040049 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:15.040052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:15.040054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:15.040502 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:15.040767 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:15.041094 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 16:06:15.041340 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":471,"threads_started":1}
I20260924 16:06:15.043280 10144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:15.135366 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 10019
I20260924 16:06:15.154525 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 9650
I20260924 16:06:15.155983 9714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:15.270463 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 9650
2026-09-24T16:06:15Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4310 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:15 dist-test-slave-1gc3 krb5kdc[10177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:15 dist-test-slave-1gc3 krb5kdc[10177](info): set up 2 sockets
Sep 24 16:06:15 dist-test-slave-1gc3 krb5kdc[10177](info): commencing operation
krb5kdc: starting...
W20260924 16:06:16.316532 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:06:16 dist-test-slave-1gc3 krb5kdc[10177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.254.keytab.
Entry for principal oryx/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.254.keytab.
I20260924 16:06:16.570307 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:43105
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.254.keytab
--principal=oryx/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:43105
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:16.930466 10191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:16.930796 10191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:16.930891 10191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:16.938665 10191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:16.938805 10191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:16.938855 10191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:16.938907 10191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:16.938969 10191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:16.949628 10191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:43105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.254.keytab
--principal=oryx/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:43105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:16.951062 10191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:16.952642 10191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:16.968395 10199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:16.968518 10197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:16.968833 10196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:16.969470 10191 server_base.cc:1034] running on GCE node
I20260924 16:06:16.970431 10191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:16.972059 10191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:16.973336 10191 hybrid_clock.cc:648] HybridClock initialized: now 1790265976973259 us; error 59 us; skew 500 ppm
I20260924 16:06:16.973791 10191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:16 dist-test-slave-1gc3 krb5kdc[10177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265976, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:16.981738 10191 init.cc:377] Logged in from keytab as oryx/127.3.214.254@KRBTEST.COM (short username oryx)
I20260924 16:06:16.983755 10191 webserver.cc:460] Webserver started at http://127.3.214.254:39721/ using document root <none> and password file <none>
I20260924 16:06:16.984527 10191 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:16.984661 10191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:16.984983 10191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:16.988323 10191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "608e51b0fef1488ba73b3f205c4758d5"
format_stamp: "Formatted at 2026-09-24 16:06:16 on dist-test-slave-1gc3"
server_key: "06f5c6d3bcbe9ade4535ef2a9cdaa16d"
server_key_iv: "fe075d4e5aa731410be7636179dc0548"
server_key_version: "encryptionkey@0"
I20260924 16:06:16.989133 10191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "608e51b0fef1488ba73b3f205c4758d5"
format_stamp: "Formatted at 2026-09-24 16:06:16 on dist-test-slave-1gc3"
server_key: "06f5c6d3bcbe9ade4535ef2a9cdaa16d"
server_key_iv: "fe075d4e5aa731410be7636179dc0548"
server_key_version: "encryptionkey@0"
I20260924 16:06:16.995210 10191 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 16:06:16.999641 10206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:17.001498 10191 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:06:17.001730 10191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "608e51b0fef1488ba73b3f205c4758d5"
format_stamp: "Formatted at 2026-09-24 16:06:16 on dist-test-slave-1gc3"
server_key: "06f5c6d3bcbe9ade4535ef2a9cdaa16d"
server_key_iv: "fe075d4e5aa731410be7636179dc0548"
server_key_version: "encryptionkey@0"
I20260924 16:06:17.001955 10191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:17.047634 10191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:17.052471 10191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:17.052765 10191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:17.073838 10191 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:43105
I20260924 16:06:17.073897 10257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:43105 every 8 connection(s)
I20260924 16:06:17.075519 10191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:17.078845 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 10191
I20260924 16:06:17.079150 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:06:17.080135 3931 external_mini_cluster.cc:1428] Setting key 2cdfecf99694b0f46f1fc500b6f08b47
I20260924 16:06:17.082206 10258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 16:06:17 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM
I20260924 16:06:17.100728 10258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5: Bootstrap starting.
I20260924 16:06:17.105502 10258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:17.107089 10258 log.cc:824] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:17.111346 10261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:17.083079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57630 (local address 127.3.214.254:43105)
0924 16:06:17.084072 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:17.084086 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:17.084140 (+ 54us) server_negotiation.cc:407] Connection header received
0924 16:06:17.086000 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:17.086027 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:17.086584 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:17.086997 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:17.089251 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:17.090946 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:17.092895 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:17.093706 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:17.093867 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:17.099684 (+ 5817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:17.099700 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:17.099732 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:17.099820 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:17.104407 (+ 4587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:17.105867 (+ 1460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:17.105881 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:17.105898 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:17.106273 (+ 375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:17.107039 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:17.107044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:17.107047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:17.108183 (+ 1136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:17.108655 (+ 472us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:17.109224 (+ 569us) server_negotiation.cc:299] Negotiation successful
0924 16:06:17.109855 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":424,"threads_started":1}
I20260924 16:06:17.111410 10258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5: No bootstrap required, opened a new log
I20260924 16:06:17.120507 10258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } }
I20260924 16:06:17.120950 10258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:17.121052 10258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e51b0fef1488ba73b3f205c4758d5, State: Initialized, Role: FOLLOWER
I20260924 16:06:17.121671 10258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } }
I20260924 16:06:17.122002 10258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:17.122193 10258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:17.122407 10258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:17.124889 10258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } }
I20260924 16:06:17.125461 10258 leader_election.cc:302] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 608e51b0fef1488ba73b3f205c4758d5; no voters:
I20260924 16:06:17.126196 10258 leader_election.cc:288] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:17.126471 10263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:17.127512 10263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [term 1 LEADER]: Becoming Leader. State: Replica: 608e51b0fef1488ba73b3f205c4758d5, State: Running, Role: LEADER
I20260924 16:06:17.128329 10263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } }
I20260924 16:06:17.128629 10258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:17.132438 10265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "608e51b0fef1488ba73b3f205c4758d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } }
I20260924 16:06:17.132552 10266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 608e51b0fef1488ba73b3f205c4758d5. Latest consensus state: current_term: 1 leader_uuid: "608e51b0fef1488ba73b3f205c4758d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } }
I20260924 16:06:17.133051 10265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:17.133049 10266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:17.137948 10274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:17.144939 10274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:17.157114 10274 catalog_manager.cc:1347] Generated new cluster ID: 1d01b2f7e31946b0b6b5616bf5533586
I20260924 16:06:17.157243 10274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:17.183722 10274 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:17.185302 10274 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:17.197326 10274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5: Generated new TSK 0
I20260924 16:06:17.198361 10274 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.214.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.214.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.214.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.193.keytab.
Entry for principal oryx/127.3.214.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.193.keytab.
I20260924 16:06:17.255686 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:43105
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.193.keytab
--principal=oryx/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:17.602896 10285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:17.603229 10285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:17.603374 10285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:17.611634 10285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:17.611773 10285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:17.612071 10285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:06:17.624212 10285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.193.keytab
--principal=oryx/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:43105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:17.625806 10285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:17.627324 10285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:17.643270 10290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:17.643232 10293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:17.643996 10291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:17.644941 10285 server_base.cc:1034] running on GCE node
I20260924 16:06:17.645879 10285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:17.647171 10285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:17.648450 10285 hybrid_clock.cc:648] HybridClock initialized: now 1790265977648341 us; error 97 us; skew 500 ppm
I20260924 16:06:17.648821 10285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:17 dist-test-slave-1gc3 krb5kdc[10177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265977, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:17.653146 10285 init.cc:377] Logged in from keytab as oryx/127.3.214.193@KRBTEST.COM (short username oryx)
I20260924 16:06:17.655416 10285 webserver.cc:460] Webserver started at http://127.3.214.193:43225/ using document root <none> and password file <none>
I20260924 16:06:17.656435 10285 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:17.656572 10285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:17.656958 10285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:17.660964 10285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "caa4aecd532a4f39aa952bd88e021aa9"
format_stamp: "Formatted at 2026-09-24 16:06:17 on dist-test-slave-1gc3"
server_key: "a398c99ec3ede6ac8043c23d57f25b09"
server_key_iv: "5e333f37d79cf298ed887a16eecd80bb"
server_key_version: "encryptionkey@0"
I20260924 16:06:17.662034 10285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "caa4aecd532a4f39aa952bd88e021aa9"
format_stamp: "Formatted at 2026-09-24 16:06:17 on dist-test-slave-1gc3"
server_key: "a398c99ec3ede6ac8043c23d57f25b09"
server_key_iv: "5e333f37d79cf298ed887a16eecd80bb"
server_key_version: "encryptionkey@0"
I20260924 16:06:17.669258 10285 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260924 16:06:17.674008 10300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:17.676080 10285 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260924 16:06:17.676335 10285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "caa4aecd532a4f39aa952bd88e021aa9"
format_stamp: "Formatted at 2026-09-24 16:06:17 on dist-test-slave-1gc3"
server_key: "a398c99ec3ede6ac8043c23d57f25b09"
server_key_iv: "5e333f37d79cf298ed887a16eecd80bb"
server_key_version: "encryptionkey@0"
I20260924 16:06:17.676590 10285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:17.699512 10285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:17.704530 10285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:17.704849 10285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:17.706259 10285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:17.708089 10285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:17.708176 10285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:17.708294 10285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:17.708356 10285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:17.745873 10285 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:37329
I20260924 16:06:17.745986 10413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:37329 every 8 connection(s)
I20260924 16:06:17.747440 10285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:06:17.749863 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 10285
I20260924 16:06:17.750067 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:06:17.750864 3931 external_mini_cluster.cc:1428] Setting key 89b2e3b4e9c7cc86aa69e8177dd87123
Sep 24 16:06:17 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265977, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM
I20260924 16:06:17.772555 10416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:17.750708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:51205 (local address 127.3.214.254:43105)
0924 16:06:17.751344 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:17.751352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:17.752599 (+ 1247us) server_negotiation.cc:407] Connection header received
0924 16:06:17.755255 (+ 2656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:17.755259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:17.755349 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:17.755469 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:17.757762 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:17.758678 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:17.760963 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:17.761434 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:17.761586 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:17.765809 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:17.765816 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:17.765821 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:17.765867 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:17.769002 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:17.769920 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:17.769926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:17.769930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:17.770078 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:17.770583 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:17.770587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:17.770590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:17.770951 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:17.771087 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:17.771898 (+ 811us) server_negotiation.cc:299] Negotiation successful
0924 16:06:17.772155 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":300,"threads_started":1}
I20260924 16:06:17.773442 10417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:17.751304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.193:51205)
0924 16:06:17.752392 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:17.752450 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:17.754858 (+ 2408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:17.755807 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:17.755819 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:17.756630 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:17.757506 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:17.757535 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:17.758875 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:17.758881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:17.760767 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:17.760786 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:17.761694 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:17.761700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:17.761940 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:17.762852 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:17.762885 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:17.765502 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:17.769201 (+ 3699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:17.769211 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:17.769228 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:17.769790 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:17.770226 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:17.770230 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:17.770232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:17.770472 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:17.771104 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:17.771111 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:17.771506 (+ 395us) client_negotiation.cc:769] Sending connection context
0924 16:06:17.771765 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 16:06:17.772217 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":428,"threads_started":1}
I20260924 16:06:17.775322 10414 heartbeater.cc:347] Connected to a master server at 127.3.214.254:43105
I20260924 16:06:17.775637 10414 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:17.776417 10414 heartbeater.cc:511] Master 127.3.214.254:43105 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.3.214.194@KRBTEST.COM; defaulting to no policy
I20260924 16:06:17.778824 10223 ts_manager.cc:194] Registered new tserver with Master: caa4aecd532a4f39aa952bd88e021aa9 (127.3.214.193:37329)
I20260924 16:06:17.780609 10223 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:51205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.214.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.214.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.194.keytab.
Entry for principal oryx/127.3.214.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.194.keytab.
I20260924 16:06:17.811439 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:43105
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.194.keytab
--principal=oryx/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:18.219061 10420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:18.219542 10420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:18.219704 10420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:18.229172 10420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:18.229393 10420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:18.229624 10420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:06:18.241788 10420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.194.keytab
--principal=oryx/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:43105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:18.243814 10420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:18.245795 10420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:18.263375 10426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:18.263787 10428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:18.263368 10425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:18.264209 10420 server_base.cc:1034] running on GCE node
I20260924 16:06:18.265116 10420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:18.266321 10420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:18.267542 10420 hybrid_clock.cc:648] HybridClock initialized: now 1790265978267458 us; error 77 us; skew 500 ppm
I20260924 16:06:18.267912 10420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:18 dist-test-slave-1gc3 krb5kdc[10177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265978, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:18.271554 10420 init.cc:377] Logged in from keytab as oryx/127.3.214.194@KRBTEST.COM (short username oryx)
I20260924 16:06:18.273725 10420 webserver.cc:460] Webserver started at http://127.3.214.194:42401/ using document root <none> and password file <none>
I20260924 16:06:18.274837 10420 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:18.274976 10420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:18.275305 10420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:18.279155 10420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af"
format_stamp: "Formatted at 2026-09-24 16:06:18 on dist-test-slave-1gc3"
server_key: "9cc433e95ebb2192b5cf82fb1dcb03d5"
server_key_iv: "6d2716e2d53a1afa4dd4ad527ea42784"
server_key_version: "encryptionkey@0"
I20260924 16:06:18.280103 10420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af"
format_stamp: "Formatted at 2026-09-24 16:06:18 on dist-test-slave-1gc3"
server_key: "9cc433e95ebb2192b5cf82fb1dcb03d5"
server_key_iv: "6d2716e2d53a1afa4dd4ad527ea42784"
server_key_version: "encryptionkey@0"
I20260924 16:06:18.287461 10420 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260924 16:06:18.292642 10435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:18.294785 10420 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:06:18.295030 10420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af"
format_stamp: "Formatted at 2026-09-24 16:06:18 on dist-test-slave-1gc3"
server_key: "9cc433e95ebb2192b5cf82fb1dcb03d5"
server_key_iv: "6d2716e2d53a1afa4dd4ad527ea42784"
server_key_version: "encryptionkey@0"
I20260924 16:06:18.295298 10420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:18.315407 10420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:18.320904 10420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:18.321239 10420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:18.322606 10420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:18.324448 10420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:18.324538 10420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:18.324667 10420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:18.324731 10420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:18.362280 10420 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:36209
I20260924 16:06:18.362255 10551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:36209 every 8 connection(s)
I20260924 16:06:18.363987 10420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:06:18.369621 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 10420
I20260924 16:06:18.369824 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:06:18.370407 3931 external_mini_cluster.cc:1428] Setting key b6ee19c374910bb89fe5a8d137e129ff
Sep 24 16:06:18 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265978, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM
I20260924 16:06:18.389655 10554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:18.367325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:42321 (local address 127.3.214.254:43105)
0924 16:06:18.367895 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:18.367901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:18.369123 (+ 1222us) server_negotiation.cc:407] Connection header received
0924 16:06:18.371771 (+ 2648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:18.371778 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:18.371894 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:18.372005 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:18.374243 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:18.375116 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:18.377002 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:18.377504 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:18.377641 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:18.382254 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:18.382261 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:18.382264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:18.382313 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:18.385565 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:18.386729 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:18.386734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:18.386737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:18.386892 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:18.387407 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:18.387410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:18.387412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:18.387787 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:18.387953 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:18.388934 (+ 981us) server_negotiation.cc:299] Negotiation successful
0924 16:06:18.389241 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":278,"threads_started":1}
I20260924 16:06:18.391033 10555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:18.367906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.194:42321)
0924 16:06:18.368858 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:18.368927 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:18.371399 (+ 2472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:18.372341 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:18.372353 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:18.373174 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:18.373967 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:18.373990 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:18.375312 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:18.375317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:18.376808 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:18.376823 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:18.377778 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:18.377785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:18.378074 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:18.378980 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:18.379009 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:18.381842 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:18.385818 (+ 3976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:18.385830 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:18.385853 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:18.386563 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:18.387016 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:18.387020 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:18.387023 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:18.387278 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:18.387935 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:18.387945 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:18.388472 (+ 527us) client_negotiation.cc:769] Sending connection context
0924 16:06:18.388782 (+ 310us) client_negotiation.cc:240] Negotiation successful
0924 16:06:18.389520 (+ 738us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":342,"threads_started":1}
WARNING: no policy specified for oryx/127.3.214.195@KRBTEST.COM; defaulting to no policy
I20260924 16:06:18.393082 10552 heartbeater.cc:347] Connected to a master server at 127.3.214.254:43105
I20260924 16:06:18.393496 10552 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.214.195@KRBTEST.COM" created.
I20260924 16:06:18.394531 10552 heartbeater.cc:511] Master 127.3.214.254:43105 requested a full tablet report, sending...
I20260924 16:06:18.396589 10223 ts_manager.cc:194] Registered new tserver with Master: acd3c54a49cd4a6ebe2cbd2a34ea80af (127.3.214.194:36209)
I20260924 16:06:18.397586 10223 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:42321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.214.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.195.keytab.
Entry for principal oryx/127.3.214.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.195.keytab.
I20260924 16:06:18.421873 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:43105
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.195.keytab
--principal=oryx/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:06:18.785106 10414 heartbeater.cc:503] Master 127.3.214.254:43105 was elected leader, sending a full tablet report...
W20260924 16:06:18.801898 10558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:18.802259 10558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:18.802362 10558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:18.810205 10558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:18.810338 10558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:18.810480 10558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:06:18.820850 10558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.195.keytab
--principal=oryx/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:43105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:18.822346 10558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:18.823808 10558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:18.839411 10563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:18.839466 10566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:18.839500 10564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:18.841288 10558 server_base.cc:1034] running on GCE node
I20260924 16:06:18.842173 10558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:18.843310 10558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:18.844581 10558 hybrid_clock.cc:648] HybridClock initialized: now 1790265978844524 us; error 47 us; skew 500 ppm
I20260924 16:06:18.844952 10558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:18 dist-test-slave-1gc3 krb5kdc[10177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265978, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:18.848595 10558 init.cc:377] Logged in from keytab as oryx/127.3.214.195@KRBTEST.COM (short username oryx)
I20260924 16:06:18.850417 10558 webserver.cc:460] Webserver started at http://127.3.214.195:36931/ using document root <none> and password file <none>
I20260924 16:06:18.851183 10558 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:18.851306 10558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:18.851620 10558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:18.854954 10558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "91edf6392c294ba1abeb6a0ffca2a21f"
format_stamp: "Formatted at 2026-09-24 16:06:18 on dist-test-slave-1gc3"
server_key: "89ac6d9c30c0890f32fbc528b8c2c3ea"
server_key_iv: "d1e7cce28d78bb3867a9ca7c62551b5d"
server_key_version: "encryptionkey@0"
I20260924 16:06:18.855844 10558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "91edf6392c294ba1abeb6a0ffca2a21f"
format_stamp: "Formatted at 2026-09-24 16:06:18 on dist-test-slave-1gc3"
server_key: "89ac6d9c30c0890f32fbc528b8c2c3ea"
server_key_iv: "d1e7cce28d78bb3867a9ca7c62551b5d"
server_key_version: "encryptionkey@0"
I20260924 16:06:18.862684 10558 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260924 16:06:18.867235 10573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:18.869122 10558 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:06:18.869360 10558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "91edf6392c294ba1abeb6a0ffca2a21f"
format_stamp: "Formatted at 2026-09-24 16:06:18 on dist-test-slave-1gc3"
server_key: "89ac6d9c30c0890f32fbc528b8c2c3ea"
server_key_iv: "d1e7cce28d78bb3867a9ca7c62551b5d"
server_key_version: "encryptionkey@0"
I20260924 16:06:18.869590 10558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:18.884852 10558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:18.889389 10558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:18.889698 10558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:18.890821 10558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:18.892324 10558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:18.892405 10558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:18.892553 10558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:18.892613 10558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:18.929811 10558 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:37151
I20260924 16:06:18.929960 10686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:37151 every 8 connection(s)
I20260924 16:06:18.931288 10558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:06:18.937630 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 10558
I20260924 16:06:18.937960 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:06:18.938647 3931 external_mini_cluster.cc:1428] Setting key a38647b61aeaa32518d1ef0292e8e9c0
Sep 24 16:06:18 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265978, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM
I20260924 16:06:18.954567 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:18.934760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:53963 (local address 127.3.214.254:43105)
0924 16:06:18.935356 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:18.935362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:18.936567 (+ 1205us) server_negotiation.cc:407] Connection header received
0924 16:06:18.938769 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:18.938775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:18.938888 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:18.938998 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:18.940830 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:18.941763 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:18.943550 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:18.943986 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:18.944105 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:18.948204 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:18.948210 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:18.948213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:18.948255 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:18.951131 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:18.951980 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:18.951984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:18.951986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:18.952120 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:18.952578 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:18.952581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:18.952583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:18.952887 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:18.953045 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:18.953850 (+ 805us) server_negotiation.cc:299] Negotiation successful
0924 16:06:18.954139 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":293,"threads_started":1}
I20260924 16:06:18.955541 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:18.935413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.195:53963)
0924 16:06:18.936347 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:18.936414 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:18.938408 (+ 1994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:18.939310 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:18.939319 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:18.939913 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:18.940605 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:18.940628 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:18.941963 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:18.941968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:18.943369 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:18.943385 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:18.944243 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:18.944251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:18.944511 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:18.945282 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:18.945308 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:18.947913 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:18.951330 (+ 3417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:18.951340 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:18.951357 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:18.951840 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:18.952235 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:18.952240 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:18.952243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:18.952457 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:18.953030 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:18.953039 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:18.953428 (+ 389us) client_negotiation.cc:769] Sending connection context
0924 16:06:18.953688 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 16:06:18.954340 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":351,"threads_started":1}
I20260924 16:06:18.957434 10687 heartbeater.cc:347] Connected to a master server at 127.3.214.254:43105
I20260924 16:06:18.957863 10687 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:18.958966 10687 heartbeater.cc:511] Master 127.3.214.254:43105 requested a full tablet report, sending...
I20260924 16:06:18.961056 10223 ts_manager.cc:194] Registered new tserver with Master: 91edf6392c294ba1abeb6a0ffca2a21f (127.3.214.195:37151)
I20260924 16:06:18.962002 10223 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:53963
I20260924 16:06:18.969791 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.214.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.214.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.214.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.253.keytab.
Entry for principal oryx/127.3.214.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.253.keytab.
I20260924 16:06:19.033919 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.253:37401
--webserver_interface=127.3.214.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.253.keytab
--principal=oryx/127.3.214.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:43105,127.3.214.253:37401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:06:19.401364 10552 heartbeater.cc:503] Master 127.3.214.254:43105 was elected leader, sending a full tablet report...
W20260924 16:06:19.474880 10693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:19.475348 10693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:19.475479 10693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:19.484319 10693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:19.484491 10693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:19.484545 10693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:19.484620 10693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:19.484673 10693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:19.496606 10693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:43105,127.3.214.253:37401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.253.keytab
--principal=oryx/127.3.214.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.253:37401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.214.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
W20260924 16:06:19.497094 10693 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.214.254:43105,127.3.214.253:37401'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 16:06:19.498559 10693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:19.500329 10693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:19.514880 10700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:19.514931 10699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:19.515381 10702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:19.516202 10693 server_base.cc:1034] running on GCE node
I20260924 16:06:19.517239 10693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:19.519176 10693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:19.520452 10693 hybrid_clock.cc:648] HybridClock initialized: now 1790265979520376 us; error 41 us; skew 500 ppm
I20260924 16:06:19.520869 10693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:19 dist-test-slave-1gc3 krb5kdc[10177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265979, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:19.524910 10693 init.cc:377] Logged in from keytab as oryx/127.3.214.253@KRBTEST.COM (short username oryx)
I20260924 16:06:19.526927 10693 webserver.cc:460] Webserver started at http://127.3.214.253:46743/ using document root <none> and password file <none>
I20260924 16:06:19.527746 10693 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:19.527848 10693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:19.528182 10693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:19.531709 10693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/data/instance:
uuid: "5849dfa252f444578e275b75c9d9e686"
format_stamp: "Formatted at 2026-09-24 16:06:19 on dist-test-slave-1gc3"
server_key: "b26d3a4bff8e4dcc5e7d9d1e80b26235"
server_key_iv: "c9bfbcea8963dc5102db1c6d3883a362"
server_key_version: "encryptionkey@0"
I20260924 16:06:19.532689 10693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/wal/instance:
uuid: "5849dfa252f444578e275b75c9d9e686"
format_stamp: "Formatted at 2026-09-24 16:06:19 on dist-test-slave-1gc3"
server_key: "b26d3a4bff8e4dcc5e7d9d1e80b26235"
server_key_iv: "c9bfbcea8963dc5102db1c6d3883a362"
server_key_version: "encryptionkey@0"
I20260924 16:06:19.539120 10693 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260924 16:06:19.543869 10709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:19.545938 10693 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260924 16:06:19.546229 10693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/wal
uuid: "5849dfa252f444578e275b75c9d9e686"
format_stamp: "Formatted at 2026-09-24 16:06:19 on dist-test-slave-1gc3"
server_key: "b26d3a4bff8e4dcc5e7d9d1e80b26235"
server_key_iv: "c9bfbcea8963dc5102db1c6d3883a362"
server_key_version: "encryptionkey@0"
I20260924 16:06:19.546550 10693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:19.574733 10693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:19.579954 10693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:19.580257 10693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:19.603720 10693 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.253:37401
I20260924 16:06:19.603785 10761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.253:37401 every 8 connection(s)
I20260924 16:06:19.606267 10693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/data/info.pb
I20260924 16:06:19.612186 10762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:19.616859 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 10693
I20260924 16:06:19.623374 10762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:06:19 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.214.253@KRBTEST.COM
Sep 24 16:06:19 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265979, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.253@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM
I20260924 16:06:19.649178 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:19.628582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57646 (local address 127.3.214.254:43105)
0924 16:06:19.630502 (+ 1920us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:19.630508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:19.630537 (+ 29us) server_negotiation.cc:407] Connection header received
0924 16:06:19.630615 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:19.630620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:19.630733 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:19.630904 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:19.632785 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.634140 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.636095 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.636886 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.637121 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:19.641449 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:19.641458 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:19.641462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:19.641523 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:19.645306 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:19.646260 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:19.646265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:19.646267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:19.646449 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:19.646944 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:19.646950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:19.646953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:19.647553 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:19.647810 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:19.648550 (+ 740us) server_negotiation.cc:299] Negotiation successful
0924 16:06:19.648851 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1753,"thread_start_us":1615,"threads_started":1}
I20260924 16:06:19.649783 10766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:19.628396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:57646)
0924 16:06:19.629858 (+ 1462us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:19.629945 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:19.630379 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:19.631356 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:19.631363 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:19.632191 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:19.632564 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:19.632580 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.634538 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.634544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:19.635835 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:19.635849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.637238 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.637244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:19.637396 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:19.638260 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:19.638293 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:19.641128 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:19.645476 (+ 4348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:19.645486 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:19.645493 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:19.646077 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:19.646591 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:19.646596 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:19.646600 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:19.646780 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:19.647831 (+ 1051us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:19.647839 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:19.648141 (+ 302us) client_negotiation.cc:769] Sending connection context
0924 16:06:19.648396 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 16:06:19.648793 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"mutex_wait_us":63,"thread_start_us":499,"threads_started":1}
I20260924 16:06:19.649806 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:19.620331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.3.214.253:37401)
0924 16:06:19.621537 (+ 1206us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:19.621557 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:19.621618 (+ 61us) server_negotiation.cc:407] Connection header received
0924 16:06:19.623820 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:19.623860 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:19.624473 (+ 613us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:19.624966 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:19.628021 (+ 3055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.630729 (+ 2708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.632803 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.634153 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.634529 (+ 376us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:19.640148 (+ 5619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:19.640168 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:19.640192 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:19.640240 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:19.643765 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:19.645082 (+ 1317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:19.645091 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:19.645101 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:19.645300 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:19.645794 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:19.645800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:19.645803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:19.646608 (+ 805us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:19.646953 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:19.647622 (+ 669us) server_negotiation.cc:299] Negotiation successful
0924 16:06:19.648160 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":869,"thread_start_us":511,"threads_started":1}
I20260924 16:06:19.652451 10762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:06:19 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265979, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.253@KRBTEST.COM for oryx/127.3.214.253@KRBTEST.COM
I20260924 16:06:19.670352 10766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:19.653669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:37401 (local address 127.0.0.1:42422)
0924 16:06:19.653962 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:19.653988 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:19.654179 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:19.654578 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:19.654581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:19.655062 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:19.655248 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:19.655257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.656347 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.656351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:19.657321 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:19.657334 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.658297 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.658302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:19.658505 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:19.659387 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:19.659412 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:19.662682 (+ 3270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:19.666954 (+ 4272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:19.666961 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:19.666965 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:19.667617 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:19.668209 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:19.668213 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:19.668216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:19.668426 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:19.669154 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:19.669159 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:19.669398 (+ 239us) client_negotiation.cc:769] Sending connection context
0924 16:06:19.669551 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 16:06:19.669980 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 16:06:19.670708 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:19.653976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42422 (local address 127.3.214.253:37401)
0924 16:06:19.654215 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:19.654223 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:19.654250 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:06:19.654327 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:19.654331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:19.654426 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:19.654532 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:19.655406 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.656184 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.657508 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.657985 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.658150 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:19.663080 (+ 4930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:19.663087 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:19.663091 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:19.663147 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:19.666653 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:19.667850 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:19.667856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:19.667859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:19.668069 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:19.668587 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:19.668592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:19.668594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:19.668982 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:19.669122 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:19.669709 (+ 587us) server_negotiation.cc:299] Negotiation successful
0924 16:06:19.670277 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":201,"server-negotiator.queue_time_us":63}
I20260924 16:06:19.677714 10762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686: Bootstrap starting.
I20260924 16:06:19.682518 10762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:19.684275 10762 log.cc:824] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:19.688289 10762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686: No bootstrap required, opened a new log
I20260924 16:06:19.696810 10762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } }
I20260924 16:06:19.697319 10762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:19.697407 10762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5849dfa252f444578e275b75c9d9e686, State: Initialized, Role: FOLLOWER
I20260924 16:06:19.698552 10762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } }
I20260924 16:06:19.698988 10769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:06:19.699457 10769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } }
I20260924 16:06:19.700659 10770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } } }
I20260924 16:06:19.701203 10770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:06:19.701856 10762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:19.701797 10769 leader_election.cc:288] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e51b0fef1488ba73b3f205c4758d5 (127.3.214.254:43105)
I20260924 16:06:19.702436 10233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5849dfa252f444578e275b75c9d9e686" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e51b0fef1488ba73b3f205c4758d5" is_pre_election: true
I20260924 16:06:19.703846 10710 leader_election.cc:302] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 5849dfa252f444578e275b75c9d9e686; no voters: 608e51b0fef1488ba73b3f205c4758d5
I20260924 16:06:19.704391 10769 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 16:06:19.713996 10781 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:06:19.714250 10781 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:06:19.717409 10693 master_runner.cc:428] Detected that this master 5849dfa252f444578e275b75c9d9e686 is joining an existing cluster
I20260924 16:06:19.717522 10693 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.214.253:37401
I20260924 16:06:19.740518 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:19.723138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57652 (local address 127.3.214.254:43105)
0924 16:06:19.723382 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:19.723390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:19.723498 (+ 108us) server_negotiation.cc:407] Connection header received
0924 16:06:19.723726 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:19.723731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:19.723803 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:19.723901 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:19.726262 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.727377 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.729973 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.730696 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.730864 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:19.734054 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:19.734060 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:19.734063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:19.734153 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:19.737054 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:19.738156 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:19.738163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:19.738166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:19.738327 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:19.738761 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:19.738765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:19.738767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:19.739126 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:19.739273 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:19.739779 (+ 506us) server_negotiation.cc:299] Negotiation successful
0924 16:06:19.740130 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260924 16:06:19.740636 10787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:19.722478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:57652)
0924 16:06:19.723359 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:19.723384 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:19.723591 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:19.723955 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:19.723959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:19.725344 (+ 1385us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:19.725973 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:19.725994 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.727583 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.727590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:19.729747 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:19.729768 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.730965 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.730977 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:19.731205 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:19.732094 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:19.732121 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:19.733757 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:19.737243 (+ 3486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:19.737254 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:19.737258 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:19.737947 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:19.738450 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:19.738456 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:19.738460 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:19.738638 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:19.739267 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:19.739272 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:19.739494 (+ 222us) client_negotiation.cc:769] Sending connection context
0924 16:06:19.739648 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 16:06:19.740053 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":74,"spinlock_wait_cycles":2944,"thread_start_us":730,"threads_started":1}
I20260924 16:06:19.742208 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:19.724437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42426 (local address 127.3.214.253:37401)
0924 16:06:19.724889 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:19.724895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:19.726043 (+ 1148us) server_negotiation.cc:407] Connection header received
0924 16:06:19.726260 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:19.726265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:19.726347 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:19.726423 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:19.727976 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.728988 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.730890 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.731490 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.731566 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:19.734090 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:19.734138 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:19.734143 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:19.734188 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:19.738316 (+ 4128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:19.739280 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:19.739286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:19.739301 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:19.739486 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:19.739939 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:19.739944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:19.739947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:19.740483 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:19.740828 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:19.741455 (+ 627us) server_negotiation.cc:299] Negotiation successful
0924 16:06:19.741741 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":70}
I20260924 16:06:19.743112 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:19.725032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:37401 (local address 127.0.0.1:42426)
0924 16:06:19.725927 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:19.725948 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:19.726090 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:19.726607 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:19.726611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:19.727418 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:19.727836 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:19.727850 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.729185 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.729195 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:19.730731 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:19.730748 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.731914 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.731922 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:19.732087 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:19.732691 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:19.732716 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:19.733884 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:19.738515 (+ 4631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:19.738525 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:19.738529 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:19.739122 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:19.739640 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:19.739645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:19.739648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:19.739816 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:19.740923 (+ 1107us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:19.740929 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:19.741128 (+ 199us) client_negotiation.cc:769] Sending connection context
0924 16:06:19.742192 (+ 1064us) client_negotiation.cc:240] Negotiation successful
0924 16:06:19.742599 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":473,"threads_started":1}
Sep 24 16:06:19 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265976, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.254@KRBTEST.COM for oryx/127.3.214.253@KRBTEST.COM
I20260924 16:06:19.770521 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:19.748874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42434 (local address 127.3.214.253:37401)
0924 16:06:19.749770 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:19.749777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:19.749807 (+ 30us) server_negotiation.cc:407] Connection header received
0924 16:06:19.750042 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:19.750050 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:19.750217 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:19.750401 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:19.752723 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.753565 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.756043 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.756751 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.756885 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:19.762294 (+ 5409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:19.762302 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:19.762307 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:19.762366 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:19.766336 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:19.767850 (+ 1514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:19.767858 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:19.767862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:19.768109 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:19.768782 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:19.768787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:19.768790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:19.769106 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:19.769254 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:19.769801 (+ 547us) server_negotiation.cc:299] Negotiation successful
0924 16:06:19.770095 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719}
I20260924 16:06:19.770589 10789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:19.748859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:37401 (local address 127.0.0.1:42434)
0924 16:06:19.749538 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:19.749588 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:19.749796 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:19.750773 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:19.750785 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:19.752049 (+ 1264us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:19.752449 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:19.752469 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.753766 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.753773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:19.755769 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:19.755792 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.757002 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.757011 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:19.757263 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:19.758220 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:19.758239 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:19.761906 (+ 3667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:19.766643 (+ 4737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:19.766675 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:19.766687 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:19.767627 (+ 940us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:19.768366 (+ 739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:19.768404 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:19.768409 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:19.768653 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:19.769260 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:19.769271 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:19.769479 (+ 208us) client_negotiation.cc:769] Sending connection context
0924 16:06:19.769672 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 16:06:19.770124 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":306,"threads_started":1}
I20260924 16:06:19.772969 10223 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.214.253:37401: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5849dfa252f444578e275b75c9d9e686"
member_type: NON_VOTER
last_known_addr {
host: "127.3.214.253"
port: 37401
}
attrs {
promote: true
}
}
dest_uuid: "608e51b0fef1488ba73b3f205c4758d5"
cas_config_opid_index: -1
I20260924 16:06:19.774137 10223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: NON_VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.3.214.252@KRBTEST.COM; defaulting to no policy
I20260924 16:06:19.792199 10736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:19.791354 10792 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5849dfa252f444578e275b75c9d9e686 (127.3.214.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: NON_VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.214.252@KRBTEST.COM" created.
I20260924 16:06:19.795717 10736 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 1 FOLLOWER]: Refusing update from remote peer 608e51b0fef1488ba73b3f205c4758d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 16:06:19.796554 10792 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.214.253:37401
I20260924 16:06:19.797379 10792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "608e51b0fef1488ba73b3f205c4758d5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: NON_VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: true } } }
I20260924 16:06:19.797878 10792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:19.804860 10793 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: NON_VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 16:06:19.807080 10693 master.cc:537] Master@127.3.214.253:37401 shutting down...
W20260924 16:06:19.810257 10209 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 16:06:19.813934 10209 consensus_peers.cc:487] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 -> Peer 5849dfa252f444578e275b75c9d9e686 (127.3.214.253:37401): Couldn't send request to peer 5849dfa252f444578e275b75c9d9e686. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 16:06:19.821395 10693 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:06:19.821938 10693 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:19.822166 10693 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.214.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.252.keytab.
Entry for principal oryx/127.3.214.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.252.keytab.
I20260924 16:06:19.840721 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.252:35111
--webserver_interface=127.3.214.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.252.keytab
--principal=oryx/127.3.214.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:43105,127.3.214.253:37401,127.3.214.252:35111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:06:19.841423 10693 master.cc:552] Master@127.3.214.253:37401 shutdown complete.
I20260924 16:06:19.841527 10693 master_runner.cc:305] Clearing existing system tablet
I20260924 16:06:19.842279 10693 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:19.842523 10693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:19.846433 10693 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:06:19.848812 10800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:19.849342 10693 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:19.849713 10693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:06:19.849812 10693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/wal
uuid: "5849dfa252f444578e275b75c9d9e686"
format_stamp: "Formatted at 2026-09-24 16:06:19 on dist-test-slave-1gc3"
server_key: "b26d3a4bff8e4dcc5e7d9d1e80b26235"
server_key_iv: "c9bfbcea8963dc5102db1c6d3883a362"
server_key_version: "encryptionkey@0"
I20260924 16:06:19.855885 10693 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:19.864274 10693 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 16:06:19.864544 10693 log.cc:1196] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 16:06:19.865042 10693 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686: Deleting consensus metadata
I20260924 16:06:19.865383 10693 master_runner.cc:315] Copying system tablet from 127.3.214.254:43105
I20260924 16:06:19.869024 10693 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686: tablet copy: Beginning tablet copy session from remote peer at address 127.3.214.254:43105
I20260924 16:06:19.891122 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:19.870302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57662 (local address 127.3.214.254:43105)
0924 16:06:19.870561 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:19.870568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:19.873966 (+ 3398us) server_negotiation.cc:407] Connection header received
0924 16:06:19.874415 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:19.874423 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:19.874539 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:19.874699 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:19.876103 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.877494 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.879587 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.880263 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.880447 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:19.883437 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:19.883445 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:19.883450 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:19.883507 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:19.887292 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:19.888451 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:19.888458 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:19.888462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:19.888672 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:19.889171 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:19.889176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:19.889179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:19.889623 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:19.889796 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:19.890424 (+ 628us) server_negotiation.cc:299] Negotiation successful
0924 16:06:19.890710 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260924 16:06:19.891218 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:19.870170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:57662)
0924 16:06:19.873732 (+ 3562us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:19.873767 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:19.874036 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:19.874766 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:19.874772 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:19.875465 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:19.875858 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:19.875873 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.877699 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.877704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:19.879378 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:19.879396 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:19.880491 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:19.880498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:19.880726 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:19.881655 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:19.881682 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:19.883124 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:19.887559 (+ 4435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:19.887567 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:19.887571 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:19.888227 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:19.888813 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:19.888818 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:19.888822 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:19.889027 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:19.889764 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:19.889770 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:19.890005 (+ 235us) client_negotiation.cc:769] Sending connection context
0924 16:06:19.890244 (+ 239us) client_negotiation.cc:240] Negotiation successful
0924 16:06:19.890708 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3362,"thread_start_us":3165,"threads_started":1}
I20260924 16:06:19.893273 10243 tablet_copy_service.cc:140] P 608e51b0fef1488ba73b3f205c4758d5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5849dfa252f444578e275b75c9d9e686 ({username='oryx', principal='oryx/127.3.214.253@KRBTEST.COM'} at 127.0.0.1:57662)
I20260924 16:06:19.893632 10243 tablet_copy_service.cc:161] P 608e51b0fef1488ba73b3f205c4758d5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5849dfa252f444578e275b75c9d9e686 at {username='oryx', principal='oryx/127.3.214.253@KRBTEST.COM'} at 127.0.0.1:57662: session id = 5849dfa252f444578e275b75c9d9e686-00000000000000000000000000000000
I20260924 16:06:19.898571 10243 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 16:06:19.902062 10693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 16:06:19.910771 10693 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686: tablet copy: Starting download of 0 data blocks...
I20260924 16:06:19.911163 10693 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686: tablet copy: Starting download of 1 WAL segments...
I20260924 16:06:19.914608 10693 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 16:06:19.919817 10243 tablet_copy_service.cc:342] P 608e51b0fef1488ba73b3f205c4758d5: Request end of tablet copy session 5849dfa252f444578e275b75c9d9e686-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.214.253@KRBTEST.COM'} at 127.0.0.1:57662
I20260924 16:06:19.920070 10243 tablet_copy_service.cc:433] P 608e51b0fef1488ba73b3f205c4758d5: ending tablet copy session 5849dfa252f444578e275b75c9d9e686-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5849dfa252f444578e275b75c9d9e686
I20260924 16:06:19.936167 10693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:19.939749 10808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:19.953732 10811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:19.958619 10809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:19.960005 10693 server_base.cc:1034] running on GCE node
I20260924 16:06:19.960482 10693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:19.961318 10693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:19.962570 10693 hybrid_clock.cc:648] HybridClock initialized: now 1790265979962492 us; error 57 us; skew 500 ppm
I20260924 16:06:19.962888 10693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:19 dist-test-slave-1gc3 krb5kdc[10177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265979, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:19.965787 10693 init.cc:377] Logged in from keytab as oryx/127.3.214.253@KRBTEST.COM (short username oryx)
I20260924 16:06:19.966058 10687 heartbeater.cc:503] Master 127.3.214.254:43105 was elected leader, sending a full tablet report...
I20260924 16:06:19.979488 10693 webserver.cc:460] Webserver started at http://127.3.214.253:43095/ using document root <none> and password file <none>
I20260924 16:06:19.980002 10693 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:19.980105 10693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:19.983851 10693 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:06:19.986168 10818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:19.986737 10693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 16:06:19.986900 10693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/wal
uuid: "5849dfa252f444578e275b75c9d9e686"
format_stamp: "Formatted at 2026-09-24 16:06:19 on dist-test-slave-1gc3"
server_key: "b26d3a4bff8e4dcc5e7d9d1e80b26235"
server_key_iv: "c9bfbcea8963dc5102db1c6d3883a362"
server_key_version: "encryptionkey@0"
I20260924 16:06:19.987099 10693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:20.045045 10693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:20.050377 10693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:20.126077 10693 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.253:37401
I20260924 16:06:20.126078 10870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.253:37401 every 8 connection(s)
I20260924 16:06:20.127125 10693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-1/data/info.pb
I20260924 16:06:20.134569 10871 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:06:20.137794 10871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686: Bootstrap starting.
I20260924 16:06:20.154425 10871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:06:20.154946 10871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686: Bootstrap complete.
I20260924 16:06:20.156178 10871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: NON_VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: true } }
I20260924 16:06:20.156464 10871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5849dfa252f444578e275b75c9d9e686, State: Initialized, Role: LEARNER
I20260924 16:06:20.156863 10871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: NON_VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: true } }
I20260924 16:06:20.157763 10874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: NON_VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: true } } }
I20260924 16:06:20.158155 10871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:20.158268 10874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [sys.catalog]: This master's current role is: LEARNER
I20260924 16:06:20.171293 10885 catalog_manager.cc:1259] Loaded cluster ID: 1d01b2f7e31946b0b6b5616bf5533586
I20260924 16:06:20.171464 10885 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686: loading cluster ID for follower catalog manager: success
I20260924 16:06:20.176460 10885 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686: acquiring CA information for follower catalog manager: success
I20260924 16:06:20.179852 10885 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:06:20.267011 10789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:20.260606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:37401 (local address 127.0.0.1:42440)
0924 16:06:20.260952 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:20.260975 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:20.261104 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:20.261981 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:20.261986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:20.262014 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:20.262242 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:20.262252 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.263512 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.263518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:20.264801 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:20.264815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.266082 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.266088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:20.266384 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:20.266461 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 16:06:20.266549 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 16:06:20.266703 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206}
I20260924 16:06:20.267069 10886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:20.260846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.3.214.253:37401)
0924 16:06:20.261619 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:20.261624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:20.261641 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:06:20.261708 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:20.261713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:20.261792 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:20.261907 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:20.262386 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.263371 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.264959 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.265906 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.266047 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:20.266383 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:20.266588 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 16:06:20.266728 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":447,"threads_started":1}
I20260924 16:06:20.268494 10846 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260924 16:06:20.269248 10874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 608e51b0fef1488ba73b3f205c4758d5. Latest consensus state: current_term: 1 leader_uuid: "608e51b0fef1488ba73b3f205c4758d5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: NON_VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: true } } }
I20260924 16:06:20.269629 10874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [sys.catalog]: This master's current role is: LEARNER
W20260924 16:06:20.349388 10799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:20.349858 10799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:20.349972 10799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:20.358853 10799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:20.359054 10799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:20.359144 10799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:20.359218 10799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:20.359282 10799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:20.372350 10799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:43105,127.3.214.253:37401,127.3.214.252:35111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.252.keytab
--principal=oryx/127.3.214.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.252:35111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.214.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:20.374584 10799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:20.376484 10799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:20.392131 10894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:20.392194 10892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:20.392208 10891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:20.393374 10799 server_base.cc:1034] running on GCE node
I20260924 16:06:20.394654 10799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:20.396721 10799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:20.398010 10799 hybrid_clock.cc:648] HybridClock initialized: now 1790265980397938 us; error 57 us; skew 500 ppm
I20260924 16:06:20.398546 10799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:20 dist-test-slave-1gc3 krb5kdc[10177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265980, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:20.410686 10799 init.cc:377] Logged in from keytab as oryx/127.3.214.252@KRBTEST.COM (short username oryx)
I20260924 16:06:20.413034 10799 webserver.cc:460] Webserver started at http://127.3.214.252:39413/ using document root <none> and password file <none>
I20260924 16:06:20.414044 10799 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:20.414247 10799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:20.414629 10799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:20.418732 10799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/data/instance:
uuid: "6fa32a0f89eb4e7ab6846d123a8f175c"
format_stamp: "Formatted at 2026-09-24 16:06:20 on dist-test-slave-1gc3"
server_key: "bbb84eea026ac527c624e2d7f74712a5"
server_key_iv: "517015117dfbc3bd770d0e0e167274e9"
server_key_version: "encryptionkey@0"
I20260924 16:06:20.419687 10799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/wal/instance:
uuid: "6fa32a0f89eb4e7ab6846d123a8f175c"
format_stamp: "Formatted at 2026-09-24 16:06:20 on dist-test-slave-1gc3"
server_key: "bbb84eea026ac527c624e2d7f74712a5"
server_key_iv: "517015117dfbc3bd770d0e0e167274e9"
server_key_version: "encryptionkey@0"
I20260924 16:06:20.426841 10799 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260924 16:06:20.431813 10901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:20.434028 10799 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:06:20.434355 10799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/wal
uuid: "6fa32a0f89eb4e7ab6846d123a8f175c"
format_stamp: "Formatted at 2026-09-24 16:06:20 on dist-test-slave-1gc3"
server_key: "bbb84eea026ac527c624e2d7f74712a5"
server_key_iv: "517015117dfbc3bd770d0e0e167274e9"
server_key_version: "encryptionkey@0"
I20260924 16:06:20.434616 10799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:20.461958 10799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:20.467262 10799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:20.467586 10799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:20.489974 10799 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.252:35111
I20260924 16:06:20.490011 10952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.252:35111 every 8 connection(s)
I20260924 16:06:20.491734 10799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/data/info.pb
I20260924 16:06:20.496292 10953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:20.501489 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 10799
I20260924 16:06:20.504915 10953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:06:20 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265980, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.252@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM
Sep 24 16:06:20 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.214.252@KRBTEST.COM
I20260924 16:06:20.530938 10957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:20.508203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:57670)
0924 16:06:20.509153 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:20.509210 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:20.509484 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:20.510523 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:20.510530 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:20.511333 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:20.511863 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:20.511885 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.513106 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.513112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:20.514420 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:20.514433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.515299 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.515306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:20.515899 (+ 593us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:20.516648 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:20.516688 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:20.520443 (+ 3755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:20.524760 (+ 4317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:20.524787 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:20.524805 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:20.525741 (+ 936us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:20.526563 (+ 822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:20.526567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:20.526570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:20.526940 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:20.527896 (+ 956us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:20.527906 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:20.528488 (+ 582us) client_negotiation.cc:769] Sending connection context
0924 16:06:20.528843 (+ 355us) client_negotiation.cc:240] Negotiation successful
0924 16:06:20.529477 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":3,"thread_start_us":360,"threads_started":1}
I20260924 16:06:20.531939 10956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:20.504388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43220 (local address 127.3.214.252:35111)
0924 16:06:20.505442 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:20.505462 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:20.505523 (+ 61us) server_negotiation.cc:407] Connection header received
0924 16:06:20.507927 (+ 2404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:20.507966 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:20.509214 (+ 1248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:20.510288 (+ 1074us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:20.511725 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.513108 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.514571 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.515106 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.515295 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:20.521518 (+ 6223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:20.521536 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:20.521564 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:20.521620 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:20.526909 (+ 5289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:20.527994 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:20.528003 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:20.528005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:20.528727 (+ 722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:20.529249 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:20.529254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:20.529257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:20.529761 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:20.530133 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:20.531217 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0924 16:06:20.531509 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":744,"thread_start_us":434,"threads_started":1}
I20260924 16:06:20.532265 10958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:20.508346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.3.214.254:43105)
0924 16:06:20.509947 (+ 1601us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:20.509955 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:20.509981 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:06:20.510057 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:20.510063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:20.510217 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:20.510359 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:20.512107 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.512945 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.514574 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.515050 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.515163 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:20.520795 (+ 5632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:20.520805 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:20.520811 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:20.520876 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:20.524496 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:20.525973 (+ 1477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:20.525983 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:20.525988 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:20.526321 (+ 333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:20.527178 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:20.527186 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:20.527189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:20.527625 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:20.527792 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:20.531560 (+ 3768us) server_negotiation.cc:299] Negotiation successful
0924 16:06:20.531892 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1431,"thread_start_us":1262,"threads_started":1}
I20260924 16:06:20.533903 10953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:06:20 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265980, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.252@KRBTEST.COM for oryx/127.3.214.253@KRBTEST.COM
I20260924 16:06:20.552904 10957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:20.535034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:37401 (local address 127.0.0.1:42452)
0924 16:06:20.535428 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:20.535449 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:20.535591 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:20.535944 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:20.535948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:20.536488 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:20.536722 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:20.536734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.537914 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.537919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:20.539509 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:20.539527 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.540521 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.540526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:20.540781 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:20.541518 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:20.541545 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:20.544875 (+ 3330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:20.549524 (+ 4649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:20.549531 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:20.549536 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:20.550241 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:20.550823 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:20.550827 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:20.550830 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:20.551013 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:20.551805 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:20.551811 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:20.552061 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 16:06:20.552244 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 16:06:20.552611 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241}
I20260924 16:06:20.553148 10886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:20.534981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42452 (local address 127.3.214.253:37401)
0924 16:06:20.535174 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:20.535179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:20.535524 (+ 345us) server_negotiation.cc:407] Connection header received
0924 16:06:20.535715 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:20.535720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:20.535788 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:20.535882 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:20.536884 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.537772 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.539701 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.540304 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.540617 (+ 313us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:20.545326 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:20.545337 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:20.545343 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:20.545413 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:20.549227 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:20.550422 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:20.550431 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:20.550434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:20.550675 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:20.551157 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:20.551163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:20.551166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:20.551644 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:20.551848 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:20.552418 (+ 570us) server_negotiation.cc:299] Negotiation successful
0924 16:06:20.552730 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:06:20.554627 10953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:06:20 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265980, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.252@KRBTEST.COM for oryx/127.3.214.252@KRBTEST.COM
I20260924 16:06:20.573984 10957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:20.555848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:35111 (local address 127.0.0.1:43234)
0924 16:06:20.556047 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:20.556065 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:20.556208 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:20.556518 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:20.556524 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:20.557089 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:20.557288 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:20.557300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.558750 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.558758 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:20.560301 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:20.560334 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.561466 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.561474 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:20.561815 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:20.562697 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:20.562725 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:20.566016 (+ 3291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:20.569815 (+ 3799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:20.569823 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:20.569829 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:20.570609 (+ 780us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:20.571375 (+ 766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:20.571383 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:20.571387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:20.571696 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:20.572671 (+ 975us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:20.572676 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:20.572902 (+ 226us) client_negotiation.cc:769] Sending connection context
0924 16:06:20.573161 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 16:06:20.573526 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 16:06:20.574322 10956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:20.555751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43234 (local address 127.3.214.252:35111)
0924 16:06:20.555949 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:20.555955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:20.556141 (+ 186us) server_negotiation.cc:407] Connection header received
0924 16:06:20.556316 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:20.556320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:20.556388 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:20.556481 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:20.557430 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.558555 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.560566 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.561216 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.561355 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:20.566421 (+ 5066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:20.566437 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:20.566443 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:20.566497 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:20.569580 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:20.570849 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:20.570856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:20.570860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:20.571151 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:20.571878 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:20.571883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:20.571887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:20.572443 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:20.572637 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:20.573285 (+ 648us) server_negotiation.cc:299] Negotiation successful
0924 16:06:20.573778 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":54}
I20260924 16:06:20.580389 10953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c: Bootstrap starting.
I20260924 16:06:20.584707 10953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:20.586361 10953 log.cc:824] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:20.590744 10953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c: No bootstrap required, opened a new log
I20260924 16:06:20.600922 10953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } }
I20260924 16:06:20.601397 10953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:20.601523 10953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa32a0f89eb4e7ab6846d123a8f175c, State: Initialized, Role: FOLLOWER
I20260924 16:06:20.602352 10953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } }
I20260924 16:06:20.603515 10960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } } }
I20260924 16:06:20.603950 10960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:06:20.604800 10953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [sys.catalog]: configured and running, proceeding with master startup.
W20260924 16:06:20.614943 10971 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:06:20.615176 10971 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:06:20.620711 10799 master_runner.cc:428] Detected that this master 6fa32a0f89eb4e7ab6846d123a8f175c is joining an existing cluster
I20260924 16:06:20.620898 10799 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.214.252:35111
I20260924 16:06:20.642714 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 10285
Sep 24 16:06:20 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.214.193@KRBTEST.COM
I20260924 16:06:20.665740 10886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:20.627182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42462 (local address 127.3.214.253:37401)
0924 16:06:20.627462 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:20.627470 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:20.647867 (+ 20397us) server_negotiation.cc:407] Connection header received
0924 16:06:20.648113 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:20.648119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:20.648211 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:20.648276 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:20.649823 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.650935 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.653186 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.653800 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.653893 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:20.657285 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:20.657294 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:20.657299 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:20.657348 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:20.661184 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:20.662844 (+ 1660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:20.662852 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:20.662856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:20.663038 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:20.663583 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:20.663588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:20.663592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:20.664026 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:20.664179 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:20.665054 (+ 875us) server_negotiation.cc:299] Negotiation successful
0924 16:06:20.665353 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 16:06:20.667063 10979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:20.627658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:37401 (local address 127.0.0.1:42462)
0924 16:06:20.647735 (+ 20077us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:20.647754 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:20.647972 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:20.648513 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:20.648518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:20.649210 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:20.649632 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:20.649644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.651120 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.651125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:20.653001 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:20.653017 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.654263 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.654269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:20.654547 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:20.655545 (+ 998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:20.655581 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:20.657005 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:20.661899 (+ 4894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:20.661905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:20.661910 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:20.662646 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:20.663213 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:20.663218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:20.663221 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:20.663439 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:20.664283 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:20.664287 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:20.664691 (+ 404us) client_negotiation.cc:769] Sending connection context
0924 16:06:20.665975 (+ 1284us) client_negotiation.cc:240] Negotiation successful
0924 16:06:20.666404 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19945,"mutex_wait_us":55,"thread_start_us":19071,"threads_started":1}
I20260924 16:06:20.669955 10958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:20.626382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.3.214.254:43105)
0924 16:06:20.626615 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:20.626621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:20.643521 (+ 16900us) server_negotiation.cc:407] Connection header received
0924 16:06:20.645081 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:20.645089 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:20.645233 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:20.645316 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:20.648102 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.649350 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.652840 (+ 3490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.653523 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.653652 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:20.660074 (+ 6422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:20.660082 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:20.660088 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:20.660146 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:20.664452 (+ 4306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:20.666971 (+ 2519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:20.666977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:20.666980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:20.667181 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:20.667683 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:20.667688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:20.667691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:20.668106 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:20.668275 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:20.669152 (+ 877us) server_negotiation.cc:299] Negotiation successful
0924 16:06:20.669468 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 16:06:20.671290 10978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:20.626654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:57678)
0924 16:06:20.643330 (+ 16676us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:20.643366 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:20.644888 (+ 1522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:20.645543 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:20.645549 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:20.646472 (+ 923us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:20.647626 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:20.647650 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.649640 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.649646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:20.651472 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:20.651485 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.653997 (+ 2512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.654003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:20.654216 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:20.656941 (+ 2725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:20.656967 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:20.658301 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:20.664668 (+ 6367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:20.664674 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:20.664677 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:20.665835 (+ 1158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:20.667357 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:20.667361 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:20.667364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:20.667545 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:20.668460 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:20.668466 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:20.668811 (+ 345us) client_negotiation.cc:769] Sending connection context
0924 16:06:20.670289 (+ 1478us) client_negotiation.cc:240] Negotiation successful
0924 16:06:20.670798 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16421,"mutex_wait_us":1,"thread_start_us":17740,"threads_started":1}
I20260924 16:06:20.672164 10956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:20.626413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43238 (local address 127.3.214.252:35111)
0924 16:06:20.626671 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:20.626677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:20.644213 (+ 17536us) server_negotiation.cc:407] Connection header received
0924 16:06:20.644451 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:20.644455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:20.644526 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:20.644618 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:20.651410 (+ 6792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.652711 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.655164 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.655894 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.655985 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:20.661133 (+ 5148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:20.661140 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:20.661145 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:20.661181 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:20.665474 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:20.669758 (+ 4284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:20.669764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:20.669767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:20.670000 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:20.670463 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:20.670468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:20.670471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:20.670914 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:20.671066 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:20.671566 (+ 500us) server_negotiation.cc:299] Negotiation successful
0924 16:06:20.671806 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":97}
I20260924 16:06:20.673529 10977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:20.626722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:35111 (local address 127.0.0.1:43238)
0924 16:06:20.644100 (+ 17378us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:20.644125 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:20.644315 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:20.645643 (+ 1328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:20.645648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:20.646646 (+ 998us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:20.647907 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:20.647921 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.652929 (+ 5008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.652934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:20.654988 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:20.655004 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.656223 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.656229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:20.656441 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:20.658558 (+ 2117us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:20.658607 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:20.659759 (+ 1152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:20.666059 (+ 6300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:20.666063 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:20.666067 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:20.666688 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:20.670184 (+ 3496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:20.670189 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:20.670193 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:20.670346 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:20.671136 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:20.671140 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:20.671290 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 16:06:20.672410 (+ 1120us) client_negotiation.cc:240] Negotiation successful
0924 16:06:20.672695 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17231,"mutex_wait_us":552,"spinlock_wait_cycles":41344,"thread_start_us":19403,"threads_started":1}
I20260924 16:06:20.677943 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:20.645149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42164 (local address 127.3.214.193:37329)
0924 16:06:20.647388 (+ 2239us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:20.647399 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:20.647435 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:06:20.647508 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:20.647517 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:20.647844 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:20.649862 (+ 2018us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:20.650173 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.651653 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.653491 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.654437 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.666178 (+ 11741us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:20.666653 (+ 475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:20.666671 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:20.666702 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:20.666745 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:20.672657 (+ 5912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:20.673942 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:20.673954 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:20.673958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:20.674255 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:20.674944 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:20.674947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:20.674949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:20.675452 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:20.676996 (+ 1544us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:20.677178 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 16:06:20.677500 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2017,"thread_start_us":1811,"threads_started":1}
I20260924 16:06:20.680137 10409 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 24 16:06:20 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265976, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.254@KRBTEST.COM for oryx/127.3.214.252@KRBTEST.COM
I20260924 16:06:20.696462 10789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:20.677688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:35111 (local address 127.0.0.1:43250)
0924 16:06:20.678078 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:20.678317 (+ 239us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:20.678443 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:20.679219 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:20.679224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:20.679664 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:20.679844 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:20.679853 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.681191 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.681198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:20.682562 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:20.682579 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.683629 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.683635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:20.683809 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:20.684511 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:20.684534 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:20.687725 (+ 3191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:20.692677 (+ 4952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:20.692684 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:20.692689 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:20.693550 (+ 861us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:20.694190 (+ 640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:20.694195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:20.694199 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:20.694386 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:20.695253 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:20.695260 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:20.695491 (+ 231us) client_negotiation.cc:769] Sending connection context
0924 16:06:20.695659 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 16:06:20.696044 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261}
I20260924 16:06:20.697844 10956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:20.678251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43250 (local address 127.3.214.252:35111)
0924 16:06:20.678652 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:20.678659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:20.678675 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:06:20.678779 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:20.678788 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:20.678909 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:20.678998 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:20.680029 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.680984 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.682713 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.683463 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.683580 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:20.688045 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:20.688054 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:20.688059 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:20.688106 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:20.692414 (+ 4308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:20.693769 (+ 1355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:20.693776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:20.693780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:20.693989 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:20.694532 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:20.694536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:20.694539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:20.695012 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:20.695175 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:20.696888 (+ 1713us) server_negotiation.cc:299] Negotiation successful
0924 16:06:20.697400 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288}
I20260924 16:06:20.700151 10223 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.214.252:35111: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c"
member_type: NON_VOTER
last_known_addr {
host: "127.3.214.252"
port: 35111
}
attrs {
promote: true
}
}
dest_uuid: "608e51b0fef1488ba73b3f205c4758d5"
cas_config_opid_index: 5
I20260924 16:06:20.701442 10223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: NON_VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: true } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: NON_VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: true } }
I20260924 16:06:20.703678 10793 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6fa32a0f89eb4e7ab6846d123a8f175c (127.3.214.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: NON_VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: true } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: NON_VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: true } } }
I20260924 16:06:20.706923 10846 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 1 LEARNER]: Refusing update from remote peer 608e51b0fef1488ba73b3f205c4758d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 16:06:20.709386 10928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:20.715837 10928 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 1 FOLLOWER]: Refusing update from remote peer 608e51b0fef1488ba73b3f205c4758d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260924 16:06:20.720291 10907 tablet.cc:2369] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:06:20.732740 10984 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: NON_VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260924 16:06:20.734336 10982 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: NON_VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:20.738355 10793 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.214.252:35111
I20260924 16:06:20.746701 10799 master.cc:537] Master@127.3.214.252:35111 shutting down...
I20260924 16:06:20.750849 10846 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6fa32a0f89eb4e7ab6846d123a8f175c (127.3.214.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: NON_VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: true } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: NON_VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: true } } }
I20260924 16:06:20.758060 10793 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5: attempting to promote NON_VOTER 5849dfa252f444578e275b75c9d9e686 to VOTER
I20260924 16:06:20.757771 10874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "608e51b0fef1488ba73b3f205c4758d5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: NON_VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: true } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: NON_VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: true } } }
I20260924 16:06:20.758248 10874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [sys.catalog]: This master's current role is: LEARNER
I20260924 16:06:20.759305 10793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: NON_VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: true } }
I20260924 16:06:20.746558 10983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "608e51b0fef1488ba73b3f205c4758d5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: NON_VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: true } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: NON_VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: true } } }
I20260924 16:06:20.760285 10983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:20.762681 10846 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 1 LEARNER]: Refusing update from remote peer 608e51b0fef1488ba73b3f205c4758d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 16:06:20.763769 10928 raft_consensus.cc:858] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: NON_VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: NON_VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: true } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: NON_VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: true } }
I20260924 16:06:20.764235 10984 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 16:06:20.767648 10982 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5849dfa252f444578e275b75c9d9e686 (127.3.214.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: NON_VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: true } } }
I20260924 16:06:20.768853 10846 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5849dfa252f444578e275b75c9d9e686 (127.3.214.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: NON_VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: true } } }
I20260924 16:06:20.772833 10982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "608e51b0fef1488ba73b3f205c4758d5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: NON_VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: true } } }
I20260924 16:06:20.773756 10874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "608e51b0fef1488ba73b3f205c4758d5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: NON_VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: true } } }
I20260924 16:06:20.774199 10874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:06:20.774442 10793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "608e51b0fef1488ba73b3f205c4758d5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: NON_VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: true } } }
I20260924 16:06:20.774739 10793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:20.773226 10982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [sys.catalog]: This master's current role is: LEADER
W20260924 16:06:20.764036 10928 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: NON_VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: NON_VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: true } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: NON_VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: true } }
W20260924 16:06:20.799763 10208 consensus_peers.cc:487] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 -> Peer 6fa32a0f89eb4e7ab6846d123a8f175c (127.3.214.252:35111): Couldn't send request to peer 6fa32a0f89eb4e7ab6846d123a8f175c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 16:06:20.805933 10928 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5849dfa252f444578e275b75c9d9e686 (127.3.214.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: NON_VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: true } } }
I20260924 16:06:20.812304 10991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "608e51b0fef1488ba73b3f205c4758d5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: NON_VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: true } } }
I20260924 16:06:20.813082 10991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260924 16:06:20.813984 10960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 608e51b0fef1488ba73b3f205c4758d5. Latest consensus state: current_term: 1 leader_uuid: "608e51b0fef1488ba73b3f205c4758d5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: NON_VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: true } } }
I20260924 16:06:20.814276 10960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260924 16:06:20.814716 10995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "608e51b0fef1488ba73b3f205c4758d5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: NON_VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: true } } }
I20260924 16:06:20.815177 10995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260924 16:06:20.816099 10799 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260924 16:06:20.816836 10799 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260924 16:06:20.817085 10799 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c: stopping tablet replica
I20260924 16:06:20.827956 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 10285
I20260924 16:06:20.836592 10799 master.cc:552] Master@127.3.214.252:35111 shutdown complete.
I20260924 16:06:20.836717 10799 master_runner.cc:305] Clearing existing system tablet
I20260924 16:06:20.837143 10799 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:20.837280 10799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:20.840749 10799 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260924 16:06:20.846692 10997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:20.847396 10799 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:20.847754 10799 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.003s
I20260924 16:06:20.847878 10799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/wal
uuid: "6fa32a0f89eb4e7ab6846d123a8f175c"
format_stamp: "Formatted at 2026-09-24 16:06:20 on dist-test-slave-1gc3"
server_key: "bbb84eea026ac527c624e2d7f74712a5"
server_key_iv: "517015117dfbc3bd770d0e0e167274e9"
server_key_version: "encryptionkey@0"
W20260924 16:06:20.851086 10210 connection.cc:535] server connection from 127.3.214.193:51205 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 16:06:20.852411 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:37329
--local_ip_for_outbound_sockets=127.3.214.193
--tserver_master_addrs=127.3.214.254:43105
--webserver_port=43225
--webserver_interface=127.3.214.193
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.193.keytab
--principal=oryx/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.214.254:43105,127.3.214.253:37401
--tserver_master_addrs=127.3.214.254:43105,127.3.214.253:37401,127.3.214.252:35111 with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:06:20.853021 10799 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:20.860790 10799 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 16:06:20.861025 10799 log.cc:1196] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 16:06:20.861657 10799 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c: Deleting consensus metadata
I20260924 16:06:20.862432 10799 master_runner.cc:315] Copying system tablet from 127.3.214.254:43105
I20260924 16:06:20.865648 10799 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c: tablet copy: Beginning tablet copy session from remote peer at address 127.3.214.254:43105
I20260924 16:06:20.883706 10958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:20.866723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57694 (local address 127.3.214.254:43105)
0924 16:06:20.866979 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:20.866986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:20.867628 (+ 642us) server_negotiation.cc:407] Connection header received
0924 16:06:20.867859 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:20.867866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:20.867997 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:20.868114 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:20.869270 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.870339 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.872294 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.873115 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.873274 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:20.876033 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:20.876040 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:20.876046 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:20.876099 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:20.880143 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:20.881158 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:20.881165 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:20.881169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:20.881363 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:20.881876 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:20.881882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:20.881886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:20.882336 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:20.882500 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:20.883003 (+ 503us) server_negotiation.cc:299] Negotiation successful
0924 16:06:20.883280 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260924 16:06:20.883913 11004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:20.866650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:57694)
0924 16:06:20.867454 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:20.867477 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:20.867665 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:20.868177 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:20.868182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:20.868799 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:20.869098 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:20.869111 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.870505 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.870511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:20.872117 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:20.872134 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:20.873325 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:20.873333 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:20.873544 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:20.874520 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:20.874546 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:20.875767 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:20.880362 (+ 4595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:20.880368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:20.880374 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:20.880995 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:20.881540 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:20.881546 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:20.881550 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:20.881745 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:20.882473 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:20.882478 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:20.882706 (+ 228us) client_negotiation.cc:769] Sending connection context
0924 16:06:20.882887 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 16:06:20.883436 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":462,"threads_started":1}
I20260924 16:06:20.884976 10243 tablet_copy_service.cc:140] P 608e51b0fef1488ba73b3f205c4758d5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6fa32a0f89eb4e7ab6846d123a8f175c ({username='oryx', principal='oryx/127.3.214.252@KRBTEST.COM'} at 127.0.0.1:57694)
I20260924 16:06:20.885182 10243 tablet_copy_service.cc:161] P 608e51b0fef1488ba73b3f205c4758d5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6fa32a0f89eb4e7ab6846d123a8f175c at {username='oryx', principal='oryx/127.3.214.252@KRBTEST.COM'} at 127.0.0.1:57694: session id = 6fa32a0f89eb4e7ab6846d123a8f175c-00000000000000000000000000000000
I20260924 16:06:20.889120 10243 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 16:06:20.891776 10799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 16:06:20.899099 10799 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c: tablet copy: Starting download of 0 data blocks...
I20260924 16:06:20.899443 10799 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c: tablet copy: Starting download of 1 WAL segments...
I20260924 16:06:20.903052 10799 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 16:06:20.909138 10243 tablet_copy_service.cc:342] P 608e51b0fef1488ba73b3f205c4758d5: Request end of tablet copy session 6fa32a0f89eb4e7ab6846d123a8f175c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.214.252@KRBTEST.COM'} at 127.0.0.1:57694
I20260924 16:06:20.909350 10243 tablet_copy_service.cc:433] P 608e51b0fef1488ba73b3f205c4758d5: ending tablet copy session 6fa32a0f89eb4e7ab6846d123a8f175c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6fa32a0f89eb4e7ab6846d123a8f175c
I20260924 16:06:20.925171 10799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:20.928270 11005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:20.934932 11006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:20.943840 11008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:20.947904 10799 server_base.cc:1034] running on GCE node
I20260924 16:06:20.948401 10799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:20.949294 10799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:20.950510 10799 hybrid_clock.cc:648] HybridClock initialized: now 1790265980950473 us; error 104 us; skew 500 ppm
I20260924 16:06:20.950814 10799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:20 dist-test-slave-1gc3 krb5kdc[10177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265980, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:20.953526 10799 init.cc:377] Logged in from keytab as oryx/127.3.214.252@KRBTEST.COM (short username oryx)
I20260924 16:06:20.959267 10799 webserver.cc:460] Webserver started at http://127.3.214.252:32893/ using document root <none> and password file <none>
I20260924 16:06:20.959692 10799 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:20.959823 10799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:20.963795 10799 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:06:20.966464 11015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:20.967125 10799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:06:20.967293 10799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/wal
uuid: "6fa32a0f89eb4e7ab6846d123a8f175c"
format_stamp: "Formatted at 2026-09-24 16:06:20 on dist-test-slave-1gc3"
server_key: "bbb84eea026ac527c624e2d7f74712a5"
server_key_iv: "517015117dfbc3bd770d0e0e167274e9"
server_key_version: "encryptionkey@0"
I20260924 16:06:20.967537 10799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:20.991003 10799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:20.996158 10799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:21.028517 10799 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.252:35111
I20260924 16:06:21.028715 11068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.252:35111 every 8 connection(s)
I20260924 16:06:21.029875 10799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/master-2/data/info.pb
I20260924 16:06:21.036113 11069 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:06:21.040988 11069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c: Bootstrap starting.
I20260924 16:06:21.067230 11069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:06:21.068102 11069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c: Bootstrap complete.
I20260924 16:06:21.069540 11069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: NON_VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: true } }
I20260924 16:06:21.069983 11069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6fa32a0f89eb4e7ab6846d123a8f175c, State: Initialized, Role: LEARNER
I20260924 16:06:21.070329 11069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: NON_VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: true } }
I20260924 16:06:21.071967 11069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:21.071703 11072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: NON_VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: true } } }
I20260924 16:06:21.072309 11072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [sys.catalog]: This master's current role is: LEARNER
I20260924 16:06:21.105971 11083 catalog_manager.cc:1259] Loaded cluster ID: 1d01b2f7e31946b0b6b5616bf5533586
I20260924 16:06:21.106207 11083 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c: loading cluster ID for follower catalog manager: success
I20260924 16:06:21.110826 11083 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c: acquiring CA information for follower catalog manager: success
I20260924 16:06:21.115293 11083 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:06:21.250923 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:21.243353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:35111 (local address 127.0.0.1:43252)
0924 16:06:21.244003 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:21.244033 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:21.244217 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:21.244928 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:21.244931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:21.244959 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:21.245212 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:21.245220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:21.247195 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:21.247199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:21.248631 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:21.248649 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:21.249994 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:21.250001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:21.250230 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:21.250268 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 16:06:21.250374 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 16:06:21.250550 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":312,"threads_started":1}
I20260924 16:06:21.250952 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:21.243505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43252 (local address 127.3.214.252:35111)
0924 16:06:21.244408 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:21.244417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:21.244450 (+ 33us) server_negotiation.cc:407] Connection header received
0924 16:06:21.244552 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:21.244559 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:21.244706 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:21.244886 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:21.245373 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:21.246998 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:21.248880 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:21.249793 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:21.249924 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:21.250243 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:21.250414 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 16:06:21.250558 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":552,"threads_started":1}
I20260924 16:06:21.252223 11043 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 16:06:21.252786 11072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 608e51b0fef1488ba73b3f205c4758d5. Latest consensus state: current_term: 1 leader_uuid: "608e51b0fef1488ba73b3f205c4758d5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: NON_VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: true } } }
I20260924 16:06:21.253115 11072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [sys.catalog]: This master's current role is: LEARNER
W20260924 16:06:21.335167 10998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:21.335553 10998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:21.335714 10998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:21.344521 10998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:21.344682 10998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:21.344817 10998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:06:21.356137 10998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.193.keytab
--principal=oryx/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:37329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=43225
--tserver_master_addrs=127.3.214.254:43105,127.3.214.253:37401,127.3.214.252:35111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:21.357949 10998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:21.359745 10998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:21.375933 11090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:21.376057 11091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:21.376025 11093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:21.377406 10998 server_base.cc:1034] running on GCE node
I20260924 16:06:21.378330 10998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:21.379619 10998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:21.380880 10998 hybrid_clock.cc:648] HybridClock initialized: now 1790265981380812 us; error 54 us; skew 500 ppm
I20260924 16:06:21.381337 10998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:21 dist-test-slave-1gc3 krb5kdc[10177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265981, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:21.385658 10998 init.cc:377] Logged in from keytab as oryx/127.3.214.193@KRBTEST.COM (short username oryx)
I20260924 16:06:21.387984 10998 webserver.cc:460] Webserver started at http://127.3.214.193:43225/ using document root <none> and password file <none>
I20260924 16:06:21.389015 10998 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:21.389202 10998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:21.396620 10998 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:06:21.400714 11100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:21.402837 10998 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260924 16:06:21.403059 10998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "caa4aecd532a4f39aa952bd88e021aa9"
format_stamp: "Formatted at 2026-09-24 16:06:17 on dist-test-slave-1gc3"
server_key: "a398c99ec3ede6ac8043c23d57f25b09"
server_key_iv: "5e333f37d79cf298ed887a16eecd80bb"
server_key_version: "encryptionkey@0"
I20260924 16:06:21.404317 10998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:21.449203 10998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:21.454466 10998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:21.454793 10998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:21.456055 10998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:21.457791 10998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:21.457885 10998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:21.457979 10998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:21.458058 10998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:21.496333 10998 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:37329
I20260924 16:06:21.496390 11213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:37329 every 8 connection(s)
I20260924 16:06:21.498430 10998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:06:21.508100 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 10998
I20260924 16:06:21.508441 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 10420
Sep 24 16:06:21 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265981, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.252@KRBTEST.COM
Sep 24 16:06:21 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265981, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM
Sep 24 16:06:21 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.214.194@KRBTEST.COM
Sep 24 16:06:21 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265981, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.253@KRBTEST.COM
I20260924 16:06:21.531016 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:21.503236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:34453 (local address 127.3.214.252:35111)
0924 16:06:21.503466 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:21.503473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:21.505089 (+ 1616us) server_negotiation.cc:407] Connection header received
0924 16:06:21.507242 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:21.507248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:21.507339 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:21.507418 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:21.511160 (+ 3742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:21.512216 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:21.516163 (+ 3947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:21.516679 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:21.516818 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:21.522397 (+ 5579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:21.522406 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:21.522411 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:21.522469 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:21.526199 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:21.527532 (+ 1333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:21.527538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:21.527542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:21.527722 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:21.528411 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:21.528417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:21.528420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:21.528837 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:21.528989 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:21.530305 (+ 1316us) server_negotiation.cc:299] Negotiation successful
0924 16:06:21.530621 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 16:06:21.531376 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:21.503483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:50039 (local address 127.3.214.254:43105)
0924 16:06:21.505539 (+ 2056us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:21.505545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:21.505566 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:06:21.511195 (+ 5629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:21.511201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:21.511310 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:21.511412 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:21.512782 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:21.513798 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:21.516042 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:21.516845 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:21.517015 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:21.523154 (+ 6139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:21.523162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:21.523167 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:21.523219 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:21.527357 (+ 4138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:21.528314 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:21.528320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:21.528323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:21.528641 (+ 318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:21.529221 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:21.529226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:21.529230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:21.529635 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:21.529783 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:21.530695 (+ 912us) server_negotiation.cc:299] Negotiation successful
0924 16:06:21.530971 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1927,"thread_start_us":1724,"threads_started":1}
I20260924 16:06:21.534124 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:21.502703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.193:50039)
0924 16:06:21.503786 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:21.503846 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:21.506744 (+ 2898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:21.511640 (+ 4896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:21.511645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:21.512106 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:21.512371 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:21.512574 (+ 203us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:21.514027 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:21.514032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:21.515911 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:21.515923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:21.517784 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:21.517793 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:21.518032 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:21.518852 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:21.518878 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:21.522842 (+ 3964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:21.527575 (+ 4733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:21.527580 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:21.527584 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:21.528153 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:21.528800 (+ 647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:21.528804 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:21.528808 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:21.528998 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:21.529873 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:21.529877 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:21.530156 (+ 279us) client_negotiation.cc:769] Sending connection context
0924 16:06:21.531675 (+ 1519us) client_negotiation.cc:240] Negotiation successful
0924 16:06:21.532395 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":505,"thread_start_us":650,"threads_started":1}
I20260924 16:06:21.534143 11219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:21.502937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:35111 (local address 127.3.214.193:34453)
0924 16:06:21.504971 (+ 2034us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:21.504996 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:21.507062 (+ 2066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:21.509171 (+ 2109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:21.509180 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:21.510131 (+ 951us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:21.510950 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:21.510972 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:21.512644 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:21.512649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:21.515695 (+ 3046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:21.515716 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:21.517015 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:21.517024 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:21.517311 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:21.518165 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:21.518235 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:21.521971 (+ 3736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:21.526478 (+ 4507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:21.526491 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:21.526525 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:21.527372 (+ 847us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:21.527890 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:21.527895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:21.527899 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:21.528272 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:21.529097 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:21.529108 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:21.529688 (+ 580us) client_negotiation.cc:769] Sending connection context
0924 16:06:21.531429 (+ 1741us) client_negotiation.cc:240] Negotiation successful
0924 16:06:21.532236 (+ 807us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1880,"mutex_wait_us":68,"thread_start_us":1629,"threads_started":1}
I20260924 16:06:21.536013 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:21.509560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33828 (local address 127.3.214.194:36209)
0924 16:06:21.510599 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:21.510611 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:21.510657 (+ 46us) server_negotiation.cc:407] Connection header received
0924 16:06:21.510777 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:21.510789 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:21.511163 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:21.511389 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:21.514207 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:21.516045 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:21.518171 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:21.519107 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:21.519264 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:21.525837 (+ 6573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:21.525865 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:21.525894 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:21.525949 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:21.530594 (+ 4645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:21.532049 (+ 1455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:21.532103 (+ 54us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:21.532108 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:21.532342 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:21.533066 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:21.533072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:21.533076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:21.533660 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:21.534017 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:21.535292 (+ 1275us) server_negotiation.cc:299] Negotiation successful
0924 16:06:21.535591 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":815,"thread_start_us":634,"threads_started":1}
I20260924 16:06:21.536124 11214 heartbeater.cc:347] Connected to a master server at 127.3.214.252:35111
I20260924 16:06:21.536166 11215 heartbeater.cc:347] Connected to a master server at 127.3.214.254:43105
I20260924 16:06:21.536556 11214 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:21.536559 11215 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:21.537868 11214 heartbeater.cc:511] Master 127.3.214.252:35111 requested a full tablet report, sending...
I20260924 16:06:21.537902 11215 heartbeater.cc:511] Master 127.3.214.254:43105 requested a full tablet report, sending...
I20260924 16:06:21.538378 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:21.506054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:37401 (local address 127.3.214.193:38697)
0924 16:06:21.506919 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:21.506938 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:21.507062 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:21.519013 (+ 11951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:21.519019 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:21.519778 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:21.520167 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:21.520182 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:21.521824 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:21.521830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:21.523395 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:21.523425 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:21.524634 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:21.524641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:21.524821 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:21.525474 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:21.525499 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:21.528619 (+ 3120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:21.533409 (+ 4790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:21.533414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:21.533418 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:21.533962 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:21.536122 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:21.536128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:21.536132 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:21.536357 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:21.537182 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:21.537188 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:21.537374 (+ 186us) client_negotiation.cc:769] Sending connection context
0924 16:06:21.537536 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 16:06:21.537923 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":3,"thread_start_us":810,"threads_started":1}
I20260924 16:06:21.538686 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:21.506741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:38697 (local address 127.3.214.253:37401)
0924 16:06:21.518523 (+ 11782us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:21.518529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:21.518564 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:06:21.518674 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:21.518680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:21.518790 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:21.519230 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:21.520370 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:21.521634 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:21.523693 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:21.524314 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:21.525111 (+ 797us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:21.528898 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:21.528905 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:21.528910 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:21.528954 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:21.533227 (+ 4273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:21.534170 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:21.534175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:21.534178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:21.534994 (+ 816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:21.536549 (+ 1555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:21.536556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:21.536560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:21.537023 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:21.537858 (+ 835us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:21.538041 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 16:06:21.538329 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9110,"thread_start_us":8980,"threads_started":1}
I20260924 16:06:21.539273 10547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:21.539723 11216 heartbeater.cc:347] Connected to a master server at 127.3.214.253:37401
I20260924 16:06:21.539882 11216 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:21.540165 10223 ts_manager.cc:194] Re-registered known tserver with Master: caa4aecd532a4f39aa952bd88e021aa9 (127.3.214.193:37329)
I20260924 16:06:21.540406 11216 heartbeater.cc:511] Master 127.3.214.253:37401 requested a full tablet report, sending...
I20260924 16:06:21.541241 10223 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:50039
I20260924 16:06:21.541425 11033 ts_manager.cc:194] Registered new tserver with Master: caa4aecd532a4f39aa952bd88e021aa9 (127.3.214.193:37329)
I20260924 16:06:21.542377 10835 ts_manager.cc:194] Registered new tserver with Master: caa4aecd532a4f39aa952bd88e021aa9 (127.3.214.193:37329)
I20260924 16:06:21.634780 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 10420
I20260924 16:06:21.654872 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:36209
--local_ip_for_outbound_sockets=127.3.214.194
--tserver_master_addrs=127.3.214.254:43105
--webserver_port=42401
--webserver_interface=127.3.214.194
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.194.keytab
--principal=oryx/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.214.254:43105,127.3.214.253:37401
--tserver_master_addrs=127.3.214.254:43105,127.3.214.253:37401,127.3.214.252:35111 with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:06:21.771428 10988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "608e51b0fef1488ba73b3f205c4758d5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: NON_VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: true } } }
I20260924 16:06:21.771817 10988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:21.773469 11227 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5: attempting to promote NON_VOTER 6fa32a0f89eb4e7ab6846d123a8f175c to VOTER
I20260924 16:06:21.774535 11227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: false } }
I20260924 16:06:21.776897 10846 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 1 FOLLOWER]: Refusing update from remote peer 608e51b0fef1488ba73b3f205c4758d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 16:06:21.777235 11043 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 1 LEARNER]: Refusing update from remote peer 608e51b0fef1488ba73b3f205c4758d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 16:06:21.777616 11227 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 16:06:21.778000 10988 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 16:06:21.781816 10988 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6fa32a0f89eb4e7ab6846d123a8f175c (127.3.214.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: false } } }
I20260924 16:06:21.782589 11043 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6fa32a0f89eb4e7ab6846d123a8f175c (127.3.214.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: false } } }
I20260924 16:06:21.786737 10846 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6fa32a0f89eb4e7ab6846d123a8f175c (127.3.214.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: false } } }
I20260924 16:06:21.786998 11233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "608e51b0fef1488ba73b3f205c4758d5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: false } } }
I20260924 16:06:21.787636 11233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:06:21.788229 11227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "608e51b0fef1488ba73b3f205c4758d5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: false } } }
I20260924 16:06:21.788584 11227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:21.788401 11228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "608e51b0fef1488ba73b3f205c4758d5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: false } } }
I20260924 16:06:21.788830 11228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:21.789336 10988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "608e51b0fef1488ba73b3f205c4758d5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: false } } }
I20260924 16:06:21.789600 10988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:21.791203 11234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "608e51b0fef1488ba73b3f205c4758d5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: false } } }
I20260924 16:06:21.791680 11234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 16:06:22.043488 11226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:22.043802 11226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:22.043898 11226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:22.051540 11226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:22.051678 11226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:22.051800 11226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:06:22.062256 11226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.194.keytab
--principal=oryx/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:36209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=42401
--tserver_master_addrs=127.3.214.254:43105,127.3.214.253:37401,127.3.214.252:35111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:22.063655 11226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:22.065029 11226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:22.082597 11240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:22.082924 11242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:22.083423 11239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:22.088040 11226 server_base.cc:1034] running on GCE node
I20260924 16:06:22.088913 11226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:22.090221 11226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:22.091501 11226 hybrid_clock.cc:648] HybridClock initialized: now 1790265982091396 us; error 98 us; skew 500 ppm
I20260924 16:06:22.091914 11226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:22 dist-test-slave-1gc3 krb5kdc[10177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265982, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:22.102398 11226 init.cc:377] Logged in from keytab as oryx/127.3.214.194@KRBTEST.COM (short username oryx)
I20260924 16:06:22.104357 11226 webserver.cc:460] Webserver started at http://127.3.214.194:42401/ using document root <none> and password file <none>
I20260924 16:06:22.105330 11226 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:22.105520 11226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:22.112735 11226 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.001s
I20260924 16:06:22.116248 11249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:22.118002 11226 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:06:22.118269 11226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af"
format_stamp: "Formatted at 2026-09-24 16:06:18 on dist-test-slave-1gc3"
server_key: "9cc433e95ebb2192b5cf82fb1dcb03d5"
server_key_iv: "6d2716e2d53a1afa4dd4ad527ea42784"
server_key_version: "encryptionkey@0"
I20260924 16:06:22.119412 11226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:22.142457 11226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:22.147251 11226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:22.147547 11226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:22.148662 11226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:22.150323 11226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:22.150424 11226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:22.150547 11226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:22.150617 11226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:22.187559 11226 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:36209
I20260924 16:06:22.187690 11362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:36209 every 8 connection(s)
I20260924 16:06:22.189248 11226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:06:22.194005 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 11226
I20260924 16:06:22.194440 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 10558
Sep 24 16:06:22 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265982, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.253@KRBTEST.COM
Sep 24 16:06:22 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265982, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.252@KRBTEST.COM
Sep 24 16:06:22 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265982, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM
Sep 24 16:06:22 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.214.195@KRBTEST.COM
I20260924 16:06:22.236249 11369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:22.194477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:41505 (local address 127.3.214.252:35111)
0924 16:06:22.209047 (+ 14570us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:22.209055 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:22.209086 (+ 31us) server_negotiation.cc:407] Connection header received
0924 16:06:22.211955 (+ 2869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:22.211963 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:22.212067 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:22.212223 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:22.215311 (+ 3088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.216524 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.219567 (+ 3043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.220012 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.220091 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:22.227339 (+ 7248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:22.227359 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:22.227363 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:22.227411 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:22.231608 (+ 4197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:22.232808 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:22.232814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:22.232818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:22.232983 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:22.233681 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:22.233688 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:22.233691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:22.234155 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:22.234325 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:22.235517 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0924 16:06:22.235803 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14427,"thread_start_us":14203,"threads_started":1}
I20260924 16:06:22.236439 11367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:22.193138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:44841 (local address 127.3.214.253:37401)
0924 16:06:22.210892 (+ 17754us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:22.210897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:22.210919 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:06:22.210989 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:22.210994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:22.211088 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:22.211213 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:22.215267 (+ 4054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.216169 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.219445 (+ 3276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.220177 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.220332 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:22.228216 (+ 7884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:22.228225 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:22.228230 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:22.228278 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:22.231916 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:22.232948 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:22.232955 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:22.232959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:22.233118 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:22.233580 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:22.233586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:22.233590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:22.234009 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:22.234219 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:22.235750 (+ 1531us) server_negotiation.cc:299] Negotiation successful
0924 16:06:22.235998 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17625,"thread_start_us":17425,"threads_started":1}
I20260924 16:06:22.237922 11370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:22.194048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:37401 (local address 127.3.214.194:44841)
0924 16:06:22.206367 (+ 12319us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:22.206405 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:22.208595 (+ 2190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:22.211617 (+ 3022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:22.211634 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:22.213100 (+ 1466us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:22.215001 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:22.215039 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.216423 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.216429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:22.219244 (+ 2815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:22.219260 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.221425 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.221439 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:22.221646 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:22.222477 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:22.222505 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:22.227157 (+ 4652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:22.232128 (+ 4971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:22.232132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:22.232136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:22.232807 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:22.233237 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:22.233243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:22.233247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:22.233467 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:22.234394 (+ 927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:22.234399 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:22.234917 (+ 518us) client_negotiation.cc:769] Sending connection context
0924 16:06:22.235281 (+ 364us) client_negotiation.cc:240] Negotiation successful
0924 16:06:22.236237 (+ 956us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10475,"mutex_wait_us":542,"spinlock_wait_cycles":63104,"thread_start_us":11446,"threads_started":1}
I20260924 16:06:22.238292 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:22.196009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:42169 (local address 127.3.214.254:43105)
0924 16:06:22.211085 (+ 15076us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:22.211092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:22.211114 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:06:22.211210 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:22.211216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:22.211321 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:22.211511 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:22.216735 (+ 5224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.217876 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.222893 (+ 5017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.223706 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.223875 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:22.227720 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:22.227728 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:22.227733 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:22.227802 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:22.231897 (+ 4095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:22.234211 (+ 2314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:22.234217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:22.234219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:22.234444 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:22.236544 (+ 2100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:22.236550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:22.236554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:22.236991 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:22.237161 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:22.237628 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 16:06:22.237893 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14946,"thread_start_us":14839,"threads_started":1}
I20260924 16:06:22.238348 11368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:22.195217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.194:42169)
0924 16:06:22.205257 (+ 10040us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:22.205338 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:22.210281 (+ 4943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:22.211617 (+ 1336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:22.211634 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:22.214053 (+ 2419us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:22.215001 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:22.215039 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.219167 (+ 4128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.219173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:22.222298 (+ 3125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:22.222316 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.224178 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.224186 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:22.224400 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:22.225043 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:22.225062 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:22.227394 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:22.233619 (+ 6225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:22.233623 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:22.233627 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:22.234027 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:22.234744 (+ 717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:22.234748 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:22.234752 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:22.234968 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:22.237154 (+ 2186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:22.237159 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:22.237316 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 16:06:22.237464 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 16:06:22.237781 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9675,"mutex_wait_us":98,"spinlock_wait_cycles":8832,"thread_start_us":9054,"threads_started":1}
I20260924 16:06:22.238945 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:22.194049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:35111 (local address 127.3.214.194:41505)
0924 16:06:22.206535 (+ 12486us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:22.206556 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:22.208590 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:22.212325 (+ 3735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:22.212331 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:22.214073 (+ 1742us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:22.215001 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:22.215043 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.216682 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.216688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:22.219331 (+ 2643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:22.219356 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.220883 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.220895 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:22.221289 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:22.222394 (+ 1105us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:22.222450 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:22.226971 (+ 4521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:22.231831 (+ 4860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:22.231844 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:22.231868 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:22.232641 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:22.233148 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:22.233153 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:22.233157 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:22.233490 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:22.234311 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:22.234321 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:22.234938 (+ 617us) client_negotiation.cc:769] Sending connection context
0924 16:06:22.235281 (+ 343us) client_negotiation.cc:240] Negotiation successful
0924 16:06:22.236038 (+ 757us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12268,"mutex_wait_us":1721,"spinlock_wait_cycles":19072,"thread_start_us":10567,"threads_started":1}
I20260924 16:06:22.240357 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:22.196711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38606 (local address 127.3.214.195:37151)
0924 16:06:22.210694 (+ 13983us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:22.210704 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:22.212681 (+ 1977us) server_negotiation.cc:407] Connection header received
0924 16:06:22.212790 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:22.212802 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:22.213154 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:22.213328 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:22.216618 (+ 3290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.218200 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.219616 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.220871 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.221035 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:22.228704 (+ 7669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:22.228728 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:22.228749 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:22.228795 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:22.233429 (+ 4634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:22.235252 (+ 1823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:22.235266 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:22.235270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:22.235485 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:22.237144 (+ 1659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:22.237150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:22.237153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:22.237728 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:22.238137 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:22.239544 (+ 1407us) server_negotiation.cc:299] Negotiation successful
0924 16:06:22.239926 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13792,"thread_start_us":13597,"threads_started":1}
I20260924 16:06:22.241497 11363 heartbeater.cc:347] Connected to a master server at 127.3.214.252:35111
I20260924 16:06:22.241523 11364 heartbeater.cc:347] Connected to a master server at 127.3.214.254:43105
I20260924 16:06:22.241921 11364 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:22.241930 11363 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:22.242236 11365 heartbeater.cc:347] Connected to a master server at 127.3.214.253:37401
I20260924 16:06:22.242364 11365 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:22.242506 10682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:22.243556 11364 heartbeater.cc:511] Master 127.3.214.254:43105 requested a full tablet report, sending...
I20260924 16:06:22.243592 11363 heartbeater.cc:511] Master 127.3.214.252:35111 requested a full tablet report, sending...
I20260924 16:06:22.243626 11365 heartbeater.cc:511] Master 127.3.214.253:37401 requested a full tablet report, sending...
I20260924 16:06:22.246037 10223 ts_manager.cc:194] Re-registered known tserver with Master: acd3c54a49cd4a6ebe2cbd2a34ea80af (127.3.214.194:36209)
I20260924 16:06:22.246186 11033 ts_manager.cc:194] Registered new tserver with Master: acd3c54a49cd4a6ebe2cbd2a34ea80af (127.3.214.194:36209)
I20260924 16:06:22.247219 10223 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:42169
I20260924 16:06:22.247637 10835 ts_manager.cc:194] Registered new tserver with Master: acd3c54a49cd4a6ebe2cbd2a34ea80af (127.3.214.194:36209)
I20260924 16:06:22.344715 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 10558
I20260924 16:06:22.364573 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:37151
--local_ip_for_outbound_sockets=127.3.214.195
--tserver_master_addrs=127.3.214.254:43105
--webserver_port=36931
--webserver_interface=127.3.214.195
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.195.keytab
--principal=oryx/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.214.254:43105,127.3.214.253:37401
--tserver_master_addrs=127.3.214.254:43105,127.3.214.253:37401,127.3.214.252:35111 with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:06:22.545084 11215 heartbeater.cc:503] Master 127.3.214.254:43105 was elected leader, sending a full tablet report...
W20260924 16:06:22.777706 11378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:22.778146 11378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:22.778307 11378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:22.786908 11378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:22.787063 11378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:22.787230 11378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:06:22.799634 11378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.195.keytab
--principal=oryx/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:37151
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=36931
--tserver_master_addrs=127.3.214.254:43105,127.3.214.253:37401,127.3.214.252:35111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:22.801635 11378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:22.803562 11378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:22.820322 11384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:22.820255 11383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:22.820255 11386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:22.821573 11378 server_base.cc:1034] running on GCE node
I20260924 16:06:22.822422 11378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:22.823640 11378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:22.824965 11378 hybrid_clock.cc:648] HybridClock initialized: now 1790265982824871 us; error 72 us; skew 500 ppm
I20260924 16:06:22.825460 11378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:22 dist-test-slave-1gc3 krb5kdc[10177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265982, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:22.829548 11378 init.cc:377] Logged in from keytab as oryx/127.3.214.195@KRBTEST.COM (short username oryx)
I20260924 16:06:22.831575 11378 webserver.cc:460] Webserver started at http://127.3.214.195:36931/ using document root <none> and password file <none>
I20260924 16:06:22.832530 11378 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:22.832686 11378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:22.840273 11378 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 16:06:22.844321 11393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:22.846390 11378 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:06:22.846634 11378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "91edf6392c294ba1abeb6a0ffca2a21f"
format_stamp: "Formatted at 2026-09-24 16:06:18 on dist-test-slave-1gc3"
server_key: "89ac6d9c30c0890f32fbc528b8c2c3ea"
server_key_iv: "d1e7cce28d78bb3867a9ca7c62551b5d"
server_key_version: "encryptionkey@0"
I20260924 16:06:22.847844 11378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:22.865304 11378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:22.869922 11378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:22.870236 11378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:22.871423 11378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:22.873271 11378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:22.873382 11378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:22.873502 11378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:22.873570 11378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:22.910859 11378 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:37151
I20260924 16:06:22.910964 11507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:37151 every 8 connection(s)
I20260924 16:06:22.912492 11378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:06:22.914330 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 11378
Sep 24 16:06:22 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265982, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM
Sep 24 16:06:22 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265982, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.253@KRBTEST.COM
Sep 24 16:06:22 dist-test-slave-1gc3 krb5kdc[10177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265982, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.252@KRBTEST.COM
I20260924 16:06:22.971583 11519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:22.921197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.195:60251)
0924 16:06:22.939570 (+ 18373us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:22.939598 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:22.939830 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:22.942625 (+ 2795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:22.942632 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:22.943233 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:22.943708 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:22.943725 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.945338 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.945345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:22.947311 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:22.947334 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.950929 (+ 3595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.950939 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:22.951406 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:22.952508 (+ 1102us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:22.952548 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:22.956555 (+ 4007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:22.964884 (+ 8329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:22.964901 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:22.964930 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:22.965847 (+ 917us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:22.966461 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:22.966467 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:22.966471 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:22.966860 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:22.967721 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:22.967734 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:22.968433 (+ 699us) client_negotiation.cc:769] Sending connection context
0924 16:06:22.968864 (+ 431us) client_negotiation.cc:240] Negotiation successful
0924 16:06:22.969604 (+ 740us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18108,"mutex_wait_us":90,"spinlock_wait_cycles":76416,"thread_start_us":18509,"threads_started":1}
I20260924 16:06:22.973618 11514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:22.920319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:60251 (local address 127.3.214.254:43105)
0924 16:06:22.937945 (+ 17626us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:22.937953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:22.939738 (+ 1785us) server_negotiation.cc:407] Connection header received
0924 16:06:22.942303 (+ 2565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:22.942311 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:22.942430 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:22.942598 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:22.943929 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.945100 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.949537 (+ 4437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.950447 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.950742 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:22.956973 (+ 6231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:22.956983 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:22.956991 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:22.957060 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:22.964592 (+ 7532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:22.966060 (+ 1468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:22.966067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:22.966071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:22.966283 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:22.967034 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:22.967040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:22.967043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:22.967517 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:22.967686 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:22.972805 (+ 5119us) server_negotiation.cc:299] Negotiation successful
0924 16:06:22.973178 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17415,"thread_start_us":19198,"threads_started":1}
I20260924 16:06:22.975425 11509 heartbeater.cc:347] Connected to a master server at 127.3.214.254:43105
I20260924 16:06:22.975869 11509 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:22.976244 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:22.925804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:37401 (local address 127.3.214.195:55253)
0924 16:06:22.940293 (+ 14489us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:22.940312 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:22.940419 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:22.941934 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:22.941943 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:22.943697 (+ 1754us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:22.945759 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:22.945778 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.947437 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.947443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:22.949269 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:22.949294 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.953261 (+ 3967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.953270 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:22.953546 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:22.954404 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:22.954448 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:22.963873 (+ 9425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:22.972437 (+ 8564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:22.972445 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:22.972449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:22.973166 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:22.973860 (+ 694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:22.973866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:22.973870 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:22.974046 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:22.974857 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:22.974862 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:22.975067 (+ 205us) client_negotiation.cc:769] Sending connection context
0924 16:06:22.975240 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 16:06:22.975636 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14347,"mutex_wait_us":29,"thread_start_us":13280,"threads_started":1}
I20260924 16:06:22.976619 11518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:22.925771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:55253 (local address 127.3.214.253:37401)
0924 16:06:22.941581 (+ 15810us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:22.941586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:22.941602 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:06:22.941657 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:22.941661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:22.941726 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:22.941830 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:22.946036 (+ 4206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.947234 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.951170 (+ 3936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.952877 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.964899 (+ 12022us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:22.965225 (+ 326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:22.965233 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:22.965238 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:22.965276 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:22.972223 (+ 6947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:22.973467 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:22.973473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:22.973476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:22.973674 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:22.974254 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:22.974259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:22.974262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:22.974695 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:22.975810 (+ 1115us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:22.975961 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 16:06:22.976242 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15694,"thread_start_us":16155,"threads_started":1}
I20260924 16:06:22.977243 11509 heartbeater.cc:511] Master 127.3.214.254:43105 requested a full tablet report, sending...
I20260924 16:06:22.978538 11510 heartbeater.cc:347] Connected to a master server at 127.3.214.253:37401
I20260924 16:06:22.978776 11510 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:22.984505 11510 heartbeater.cc:511] Master 127.3.214.253:37401 requested a full tablet report, sending...
I20260924 16:06:22.988106 10835 ts_manager.cc:194] Registered new tserver with Master: 91edf6392c294ba1abeb6a0ffca2a21f (127.3.214.195:37151)
I20260924 16:06:22.988893 10223 ts_manager.cc:194] Re-registered known tserver with Master: 91edf6392c294ba1abeb6a0ffca2a21f (127.3.214.195:37151)
I20260924 16:06:22.989440 11515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:22.920319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:39629 (local address 127.3.214.252:35111)
0924 16:06:22.938423 (+ 18104us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:22.938431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:22.938458 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:06:22.938560 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:22.938567 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:22.938852 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:22.939080 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:22.942485 (+ 3405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.943739 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.953872 (+ 10133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.954645 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.954772 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:22.970592 (+ 15820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:22.970601 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:22.970606 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:22.970663 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:22.975399 (+ 4736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:22.977401 (+ 2002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:22.977409 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:22.977414 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:22.977667 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:22.987222 (+ 9555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:22.987230 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:22.987234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:22.987780 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:22.987973 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:22.988700 (+ 727us) server_negotiation.cc:299] Negotiation successful
0924 16:06:22.989054 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17955,"thread_start_us":17832,"threads_started":1}
I20260924 16:06:22.990688 11516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:22.921196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:35111 (local address 127.3.214.195:39629)
0924 16:06:22.930932 (+ 9736us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:22.931041 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:22.936585 (+ 5544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:22.940386 (+ 3801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:22.940401 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:22.941216 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:22.942141 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:22.942176 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.943985 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.943991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:22.953463 (+ 9472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:22.953481 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.964804 (+ 11323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.964812 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:22.965073 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:22.966516 (+ 1443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:22.966542 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:22.970204 (+ 3662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:22.975604 (+ 5400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:22.975609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:22.975612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:22.977155 (+ 1543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:22.986660 (+ 9505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:22.986667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:22.986672 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:22.987010 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:22.988094 (+ 1084us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:22.988099 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:22.988304 (+ 205us) client_negotiation.cc:769] Sending connection context
0924 16:06:22.989744 (+ 1440us) client_negotiation.cc:240] Negotiation successful
0924 16:06:22.990233 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9211,"mutex_wait_us":3,"thread_start_us":8846,"threads_started":1}
I20260924 16:06:22.991837 11508 heartbeater.cc:347] Connected to a master server at 127.3.214.252:35111
I20260924 16:06:22.992103 11508 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:22.992046 10223 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:60251
I20260924 16:06:22.992709 11508 heartbeater.cc:511] Master 127.3.214.252:35111 requested a full tablet report, sending...
I20260924 16:06:22.994343 11033 ts_manager.cc:194] Registered new tserver with Master: 91edf6392c294ba1abeb6a0ffca2a21f (127.3.214.195:37151)
I20260924 16:06:23.003532 11514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:22.947582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57710 (local address 127.3.214.254:43105)
0924 16:06:22.973909 (+ 26327us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:22.973915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:22.973941 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:06:22.974029 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:22.974035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:22.974167 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:22.974320 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:22.980878 (+ 6558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.982296 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.984497 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.985189 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.985337 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:22.993771 (+ 8434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:22.993779 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:22.993783 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:22.993835 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:22.998352 (+ 4517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:23.000404 (+ 2052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:23.000420 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:23.000425 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:23.000624 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:23.001192 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:23.001198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:23.001201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:23.001695 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:23.002018 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:23.002659 (+ 641us) server_negotiation.cc:299] Negotiation successful
0924 16:06:23.003144 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26211,"thread_start_us":38797,"threads_started":1}
I20260924 16:06:23.009925 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:22.955279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43260 (local address 127.3.214.252:35111)
0924 16:06:22.967287 (+ 12008us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:22.967294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:22.990758 (+ 23464us) server_negotiation.cc:407] Connection header received
0924 16:06:22.990935 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:22.990945 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:22.991076 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:22.991210 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:22.995015 (+ 3805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.996180 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.998211 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.998747 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.998828 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:23.002753 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:23.002760 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:23.002765 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:23.002810 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:23.006425 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:23.007443 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:23.007448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:23.007451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:23.007627 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:23.008251 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:23.008255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:23.008257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:23.008565 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:23.008807 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:23.009266 (+ 459us) server_negotiation.cc:299] Negotiation successful
0924 16:06:23.009524 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11058,"thread_start_us":10897,"threads_started":1}
I20260924 16:06:23.010787 11524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:22.948609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42468 (local address 127.3.214.253:37401)
0924 16:06:22.964662 (+ 16053us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:22.964670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:22.989160 (+ 24490us) server_negotiation.cc:407] Connection header received
0924 16:06:22.989249 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:22.989256 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:22.989351 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:22.989460 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:22.993822 (+ 4362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.995281 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.997324 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:22.997847 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:22.997987 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:23.002346 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:23.002357 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:23.002362 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:23.002420 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:23.005820 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:23.006866 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:23.006874 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:23.006877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:23.007146 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:23.007603 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:23.007610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:23.007614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:23.008073 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:23.008378 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:23.009595 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0924 16:06:23.009897 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15876,"thread_start_us":18386,"threads_started":1}
I20260924 16:06:23.009218 10223 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:06:23.018586 10223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:06:23.066421 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:23.052074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:37329 (local address 127.0.0.1:42176)
0924 16:06:23.055730 (+ 3656us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:23.055753 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:23.056009 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:23.057129 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:23.057135 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:23.057167 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:23.057597 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:23.057610 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.060268 (+ 2658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.060275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:23.062915 (+ 2640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:23.062937 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.065447 (+ 2510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.065458 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:23.065631 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:23.065664 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 16:06:23.065755 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 16:06:23.065925 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3506,"thread_start_us":3356,"threads_started":1}
I20260924 16:06:23.068075 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:23.052884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42176 (local address 127.3.214.193:37329)
0924 16:06:23.055254 (+ 2370us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:23.055260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:23.055871 (+ 611us) server_negotiation.cc:407] Connection header received
0924 16:06:23.056240 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:23.056250 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:23.056614 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:23.056812 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:23.057811 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.060025 (+ 2214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.063147 (+ 3122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.064895 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.065005 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:23.065236 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:23.067091 (+ 1855us) server_negotiation.cc:299] Negotiation successful
0924 16:06:23.067280 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2216,"thread_start_us":2072,"threads_started":1}
I20260924 16:06:23.087114 11149 tablet_service.cc:1467] Processing CreateTablet for tablet a5d6d6c961504f3aba94819ef4a13123 (DEFAULT_TABLE table=test-table [id=cf71fe47095a4808bbc43e601fa5ef4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:23.089607 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:23.053551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:37151 (local address 127.0.0.1:38608)
0924 16:06:23.066729 (+ 13178us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:23.066753 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:23.066887 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:23.068554 (+ 1667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:23.068563 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:23.068606 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:23.068833 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:23.068846 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.071358 (+ 2512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.071365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:23.086256 (+ 14891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:23.086283 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.088767 (+ 2484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.088774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:23.088993 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:23.089030 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 16:06:23.089119 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 16:06:23.089288 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13057,"thread_start_us":29760,"threads_started":1}
I20260924 16:06:23.090238 11149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5d6d6c961504f3aba94819ef4a13123. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:23.090530 11542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:23.054175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38608 (local address 127.3.214.195:37151)
0924 16:06:23.055002 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:23.055012 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:23.066877 (+ 11865us) server_negotiation.cc:407] Connection header received
0924 16:06:23.067043 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:23.067052 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:23.067366 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:23.067586 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:23.069094 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.071097 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.086561 (+ 15464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.088274 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.088417 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:23.088647 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:23.089982 (+ 1335us) server_negotiation.cc:299] Negotiation successful
0924 16:06:23.090192 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":391,"threads_started":1}
I20260924 16:06:23.093357 11544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:23.055914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:36209 (local address 127.0.0.1:33832)
0924 16:06:23.079415 (+ 23501us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:23.079449 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:23.079706 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:23.084835 (+ 5129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:23.084843 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:23.084880 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:23.085374 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:23.085391 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.087577 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.087584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:23.090231 (+ 2647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:23.090254 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.092364 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.092373 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:23.092601 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:23.092645 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 16:06:23.092732 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 16:06:23.092930 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23258,"thread_start_us":23951,"threads_started":1}
I20260924 16:06:23.095172 11545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:23.056187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33832 (local address 127.3.214.194:36209)
0924 16:06:23.084008 (+ 27821us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:23.084019 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:23.084059 (+ 40us) server_negotiation.cc:407] Connection header received
0924 16:06:23.084200 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:23.084210 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:23.084543 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:23.085616 (+ 1073us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:23.085884 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.087389 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.090522 (+ 3133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.092038 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.094216 (+ 2178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:23.094501 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:23.094597 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 16:06:23.094839 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27570,"thread_start_us":27436,"threads_started":1}
I20260924 16:06:23.094905 11441 tablet_service.cc:1467] Processing CreateTablet for tablet a5d6d6c961504f3aba94819ef4a13123 (DEFAULT_TABLE table=test-table [id=cf71fe47095a4808bbc43e601fa5ef4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:23.097198 11441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5d6d6c961504f3aba94819ef4a13123. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:23.100592 11297 tablet_service.cc:1467] Processing CreateTablet for tablet a5d6d6c961504f3aba94819ef4a13123 (DEFAULT_TABLE table=test-table [id=cf71fe47095a4808bbc43e601fa5ef4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:23.102728 11297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5d6d6c961504f3aba94819ef4a13123. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:23.111956 11546 tablet_bootstrap.cc:492] T a5d6d6c961504f3aba94819ef4a13123 P 91edf6392c294ba1abeb6a0ffca2a21f: Bootstrap starting.
I20260924 16:06:23.113554 11547 tablet_bootstrap.cc:492] T a5d6d6c961504f3aba94819ef4a13123 P caa4aecd532a4f39aa952bd88e021aa9: Bootstrap starting.
I20260924 16:06:23.116237 11546 tablet_bootstrap.cc:654] T a5d6d6c961504f3aba94819ef4a13123 P 91edf6392c294ba1abeb6a0ffca2a21f: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:23.117652 11547 tablet_bootstrap.cc:654] T a5d6d6c961504f3aba94819ef4a13123 P caa4aecd532a4f39aa952bd88e021aa9: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:23.117764 11546 log.cc:824] T a5d6d6c961504f3aba94819ef4a13123 P 91edf6392c294ba1abeb6a0ffca2a21f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:23.119139 11547 log.cc:824] T a5d6d6c961504f3aba94819ef4a13123 P caa4aecd532a4f39aa952bd88e021aa9: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:23.122340 11547 tablet_bootstrap.cc:492] T a5d6d6c961504f3aba94819ef4a13123 P caa4aecd532a4f39aa952bd88e021aa9: No bootstrap required, opened a new log
I20260924 16:06:23.122350 11546 tablet_bootstrap.cc:492] T a5d6d6c961504f3aba94819ef4a13123 P 91edf6392c294ba1abeb6a0ffca2a21f: No bootstrap required, opened a new log
I20260924 16:06:23.122658 11547 ts_tablet_manager.cc:1397] T a5d6d6c961504f3aba94819ef4a13123 P caa4aecd532a4f39aa952bd88e021aa9: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260924 16:06:23.122718 11546 ts_tablet_manager.cc:1397] T a5d6d6c961504f3aba94819ef4a13123 P 91edf6392c294ba1abeb6a0ffca2a21f: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260924 16:06:23.123833 11550 tablet_bootstrap.cc:492] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af: Bootstrap starting.
I20260924 16:06:23.129668 11550 tablet_bootstrap.cc:654] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:23.131759 11550 log.cc:824] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:23.134228 11547 raft_consensus.cc:348] T a5d6d6c961504f3aba94819ef4a13123 P caa4aecd532a4f39aa952bd88e021aa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa4aecd532a4f39aa952bd88e021aa9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37329 } } peers { permanent_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 36209 } } peers { permanent_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37151 } }
I20260924 16:06:23.134871 11547 raft_consensus.cc:374] T a5d6d6c961504f3aba94819ef4a13123 P caa4aecd532a4f39aa952bd88e021aa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:23.135010 11547 raft_consensus.cc:729] T a5d6d6c961504f3aba94819ef4a13123 P caa4aecd532a4f39aa952bd88e021aa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caa4aecd532a4f39aa952bd88e021aa9, State: Initialized, Role: FOLLOWER
I20260924 16:06:23.135943 11547 consensus_queue.cc:260] T a5d6d6c961504f3aba94819ef4a13123 P caa4aecd532a4f39aa952bd88e021aa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa4aecd532a4f39aa952bd88e021aa9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37329 } } peers { permanent_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 36209 } } peers { permanent_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37151 } }
I20260924 16:06:23.136328 11546 raft_consensus.cc:348] T a5d6d6c961504f3aba94819ef4a13123 P 91edf6392c294ba1abeb6a0ffca2a21f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa4aecd532a4f39aa952bd88e021aa9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37329 } } peers { permanent_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 36209 } } peers { permanent_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37151 } }
I20260924 16:06:23.137030 11546 raft_consensus.cc:374] T a5d6d6c961504f3aba94819ef4a13123 P 91edf6392c294ba1abeb6a0ffca2a21f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:23.137179 11546 raft_consensus.cc:729] T a5d6d6c961504f3aba94819ef4a13123 P 91edf6392c294ba1abeb6a0ffca2a21f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91edf6392c294ba1abeb6a0ffca2a21f, State: Initialized, Role: FOLLOWER
I20260924 16:06:23.138176 11546 consensus_queue.cc:260] T a5d6d6c961504f3aba94819ef4a13123 P 91edf6392c294ba1abeb6a0ffca2a21f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa4aecd532a4f39aa952bd88e021aa9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37329 } } peers { permanent_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 36209 } } peers { permanent_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37151 } }
I20260924 16:06:23.142915 11550 tablet_bootstrap.cc:492] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af: No bootstrap required, opened a new log
I20260924 16:06:23.143443 11550 ts_tablet_manager.cc:1397] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.003s
I20260924 16:06:23.148962 11546 ts_tablet_manager.cc:1428] T a5d6d6c961504f3aba94819ef4a13123 P 91edf6392c294ba1abeb6a0ffca2a21f: Time spent starting tablet: real 0.026s user 0.010s sys 0.012s
I20260924 16:06:23.149435 11509 heartbeater.cc:503] Master 127.3.214.254:43105 was elected leader, sending a full tablet report...
I20260924 16:06:23.150933 11547 ts_tablet_manager.cc:1428] T a5d6d6c961504f3aba94819ef4a13123 P caa4aecd532a4f39aa952bd88e021aa9: Time spent starting tablet: real 0.028s user 0.023s sys 0.004s
I20260924 16:06:23.159722 11550 raft_consensus.cc:348] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa4aecd532a4f39aa952bd88e021aa9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37329 } } peers { permanent_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 36209 } } peers { permanent_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37151 } }
I20260924 16:06:23.160259 11550 raft_consensus.cc:374] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:23.160382 11550 raft_consensus.cc:729] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd3c54a49cd4a6ebe2cbd2a34ea80af, State: Initialized, Role: FOLLOWER
I20260924 16:06:23.161074 11550 consensus_queue.cc:260] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa4aecd532a4f39aa952bd88e021aa9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37329 } } peers { permanent_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 36209 } } peers { permanent_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37151 } }
I20260924 16:06:23.162436 11364 heartbeater.cc:503] Master 127.3.214.254:43105 was elected leader, sending a full tablet report...
I20260924 16:06:23.163182 11550 ts_tablet_manager.cc:1428] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af: Time spent starting tablet: real 0.020s user 0.010s sys 0.004s
W20260924 16:06:23.176702 11511 tablet.cc:2369] T a5d6d6c961504f3aba94819ef4a13123 P 91edf6392c294ba1abeb6a0ffca2a21f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:06:23.206574 11554 raft_consensus.cc:482] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:06:23.206966 11554 raft_consensus.cc:504] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa4aecd532a4f39aa952bd88e021aa9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37329 } } peers { permanent_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 36209 } } peers { permanent_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37151 } }
I20260924 16:06:23.209227 11554 leader_election.cc:288] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers caa4aecd532a4f39aa952bd88e021aa9 (127.3.214.193:37329), 91edf6392c294ba1abeb6a0ffca2a21f (127.3.214.195:37151)
W20260924 16:06:23.214931 11366 tablet.cc:2369] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:06:23.217784 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:23.209615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:44081 (local address 127.3.214.193:37329)
0924 16:06:23.209855 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:23.209862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:23.210822 (+ 960us) server_negotiation.cc:407] Connection header received
0924 16:06:23.211046 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:23.211051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:23.211118 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:23.211190 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:23.211982 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.212799 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.215249 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.216604 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.216795 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:23.216892 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:23.217278 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 16:06:23.217477 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 16:06:23.217832 11542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:23.209868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:51485 (local address 127.3.214.195:37151)
0924 16:06:23.210168 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:23.210176 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:23.210894 (+ 718us) server_negotiation.cc:407] Connection header received
0924 16:06:23.211106 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:23.211112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:23.211189 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:23.211297 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:23.212046 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.212848 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.215453 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.216742 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.216907 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:23.216976 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:23.217429 (+ 453us) server_negotiation.cc:299] Negotiation successful
0924 16:06:23.217571 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 16:06:23.217923 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:23.209565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:37329 (local address 127.3.214.194:44081)
0924 16:06:23.210671 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:23.210696 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:23.210884 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:23.211240 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:23.211245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:23.211273 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:23.211825 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:23.211840 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.212972 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.212978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:23.215056 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:23.215076 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.216861 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.216866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:23.217042 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:23.217080 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 16:06:23.217186 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 16:06:23.217357 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":37,"spinlock_wait_cycles":1920,"thread_start_us":600,"threads_started":1}
I20260924 16:06:23.218050 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:23.209654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:37151 (local address 127.3.214.194:51485)
0924 16:06:23.210735 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:23.210757 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:23.210942 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:23.211338 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:23.211341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:23.211369 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:23.211856 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:23.211868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.212996 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.213000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:23.215182 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:23.215205 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.217103 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.217109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:23.217245 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:23.217277 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 16:06:23.217372 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 16:06:23.217538 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"mutex_wait_us":65,"thread_start_us":782,"threads_started":1}
I20260924 16:06:23.219383 11462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d6d6c961504f3aba94819ef4a13123" candidate_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" is_pre_election: true
I20260924 16:06:23.219569 11169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d6d6c961504f3aba94819ef4a13123" candidate_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caa4aecd532a4f39aa952bd88e021aa9" is_pre_election: true
I20260924 16:06:23.219981 11462 raft_consensus.cc:2454] T a5d6d6c961504f3aba94819ef4a13123 P 91edf6392c294ba1abeb6a0ffca2a21f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acd3c54a49cd4a6ebe2cbd2a34ea80af in term 0.
I20260924 16:06:23.220183 11169 raft_consensus.cc:2454] T a5d6d6c961504f3aba94819ef4a13123 P caa4aecd532a4f39aa952bd88e021aa9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acd3c54a49cd4a6ebe2cbd2a34ea80af in term 0.
I20260924 16:06:23.220778 11251 leader_election.cc:302] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91edf6392c294ba1abeb6a0ffca2a21f, acd3c54a49cd4a6ebe2cbd2a34ea80af; no voters:
I20260924 16:06:23.221263 11554 raft_consensus.cc:2781] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:06:23.221432 11554 raft_consensus.cc:482] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:06:23.221545 11554 raft_consensus.cc:3037] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:23.224283 11554 raft_consensus.cc:504] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa4aecd532a4f39aa952bd88e021aa9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37329 } } peers { permanent_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 36209 } } peers { permanent_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37151 } }
I20260924 16:06:23.225044 11554 leader_election.cc:288] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af [CANDIDATE]: Term 1 election: Requested vote from peers caa4aecd532a4f39aa952bd88e021aa9 (127.3.214.193:37329), 91edf6392c294ba1abeb6a0ffca2a21f (127.3.214.195:37151)
I20260924 16:06:23.225685 11169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d6d6c961504f3aba94819ef4a13123" candidate_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caa4aecd532a4f39aa952bd88e021aa9"
I20260924 16:06:23.225785 11462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5d6d6c961504f3aba94819ef4a13123" candidate_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91edf6392c294ba1abeb6a0ffca2a21f"
I20260924 16:06:23.225920 11169 raft_consensus.cc:3037] T a5d6d6c961504f3aba94819ef4a13123 P caa4aecd532a4f39aa952bd88e021aa9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:23.226063 11462 raft_consensus.cc:3037] T a5d6d6c961504f3aba94819ef4a13123 P 91edf6392c294ba1abeb6a0ffca2a21f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:23.228580 11169 raft_consensus.cc:2454] T a5d6d6c961504f3aba94819ef4a13123 P caa4aecd532a4f39aa952bd88e021aa9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd3c54a49cd4a6ebe2cbd2a34ea80af in term 1.
I20260924 16:06:23.228610 11462 raft_consensus.cc:2454] T a5d6d6c961504f3aba94819ef4a13123 P 91edf6392c294ba1abeb6a0ffca2a21f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd3c54a49cd4a6ebe2cbd2a34ea80af in term 1.
I20260924 16:06:23.229185 11252 leader_election.cc:302] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 91edf6392c294ba1abeb6a0ffca2a21f, acd3c54a49cd4a6ebe2cbd2a34ea80af, caa4aecd532a4f39aa952bd88e021aa9; no voters:
I20260924 16:06:23.229625 11554 raft_consensus.cc:2781] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:23.230253 11554 raft_consensus.cc:686] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af [term 1 LEADER]: Becoming Leader. State: Replica: acd3c54a49cd4a6ebe2cbd2a34ea80af, State: Running, Role: LEADER
I20260924 16:06:23.230959 11554 consensus_queue.cc:237] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa4aecd532a4f39aa952bd88e021aa9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37329 } } peers { permanent_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 36209 } } peers { permanent_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37151 } }
I20260924 16:06:23.237210 10222 catalog_manager.cc:5495] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af reported cstate change: term changed from 0 to 1, leader changed from <none> to acd3c54a49cd4a6ebe2cbd2a34ea80af (127.3.214.194). New cstate: current_term: 1 leader_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa4aecd532a4f39aa952bd88e021aa9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 36209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37151 } health_report { overall_health: UNKNOWN } } }
W20260924 16:06:23.258852 11217 tablet.cc:2369] T a5d6d6c961504f3aba94819ef4a13123 P caa4aecd532a4f39aa952bd88e021aa9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:06:23.303320 11545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:23.297247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33844 (local address 127.3.214.194:36209)
0924 16:06:23.297507 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:23.297514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:23.297536 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:06:23.297675 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:23.297680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:23.297779 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:23.297944 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:23.298437 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.299263 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.300716 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.301158 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.301272 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:23.301877 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:23.302643 (+ 766us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:23.302794 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:23.302902 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 16:06:23.303002 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 16:06:23.314196 11169 raft_consensus.cc:1261] T a5d6d6c961504f3aba94819ef4a13123 P caa4aecd532a4f39aa952bd88e021aa9 [term 1 FOLLOWER]: Refusing update from remote peer acd3c54a49cd4a6ebe2cbd2a34ea80af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:23.314252 11462 raft_consensus.cc:1261] T a5d6d6c961504f3aba94819ef4a13123 P 91edf6392c294ba1abeb6a0ffca2a21f [term 1 FOLLOWER]: Refusing update from remote peer acd3c54a49cd4a6ebe2cbd2a34ea80af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:23.315609 11554 consensus_queue.cc:1035] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af [LEADER]: Connected to new peer: Peer: permanent_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:23.316039 11557 consensus_queue.cc:1035] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af [LEADER]: Connected to new peer: Peer: permanent_uuid: "caa4aecd532a4f39aa952bd88e021aa9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:23.338474 11562 mvcc.cc:205] Tried to move back new op lower bound from 7332929467640279040 to 7332929467318968320. Current Snapshot: MvccSnapshot[applied={T|T < 7332929467640279040}]
I20260924 16:06:23.355585 10222 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:23.355855 10222 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:23.363655 10222 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 608e51b0fef1488ba73b3f205c4758d5: Sending DeleteTablet for 3 replicas of tablet a5d6d6c961504f3aba94819ef4a13123
I20260924 16:06:23.365366 11297 tablet_service.cc:1514] Processing DeleteTablet for tablet a5d6d6c961504f3aba94819ef4a13123 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:23 UTC) from {username='oryx', principal='oryx/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:33832
I20260924 16:06:23.365749 11441 tablet_service.cc:1514] Processing DeleteTablet for tablet a5d6d6c961504f3aba94819ef4a13123 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:23 UTC) from {username='oryx', principal='oryx/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:38608
I20260924 16:06:23.366477 11149 tablet_service.cc:1514] Processing DeleteTablet for tablet a5d6d6c961504f3aba94819ef4a13123 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:23 UTC) from {username='oryx', principal='oryx/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:42176
I20260924 16:06:23.366824 11571 tablet_replica.cc:330] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af: stopping tablet replica
I20260924 16:06:23.367430 11571 raft_consensus.cc:2229] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:06:23.368403 11571 raft_consensus.cc:2258] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:23.371254 11571 ts_tablet_manager.cc:1905] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:23.381464 11571 ts_tablet_manager.cc:1918] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:06:23.382172 11572 tablet_replica.cc:330] T a5d6d6c961504f3aba94819ef4a13123 P 91edf6392c294ba1abeb6a0ffca2a21f: stopping tablet replica
I20260924 16:06:23.382635 11572 raft_consensus.cc:2229] T a5d6d6c961504f3aba94819ef4a13123 P 91edf6392c294ba1abeb6a0ffca2a21f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:06:23.383147 11572 raft_consensus.cc:2258] T a5d6d6c961504f3aba94819ef4a13123 P 91edf6392c294ba1abeb6a0ffca2a21f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:23.383672 11571 log.cc:1196] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/a5d6d6c961504f3aba94819ef4a13123
I20260924 16:06:23.384298 11571 ts_tablet_manager.cc:1939] T a5d6d6c961504f3aba94819ef4a13123 P acd3c54a49cd4a6ebe2cbd2a34ea80af: Deleting consensus metadata
I20260924 16:06:23.385440 11572 ts_tablet_manager.cc:1905] T a5d6d6c961504f3aba94819ef4a13123 P 91edf6392c294ba1abeb6a0ffca2a21f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:23.385456 11573 tablet_replica.cc:330] T a5d6d6c961504f3aba94819ef4a13123 P caa4aecd532a4f39aa952bd88e021aa9: stopping tablet replica
I20260924 16:06:23.385972 11573 raft_consensus.cc:2229] T a5d6d6c961504f3aba94819ef4a13123 P caa4aecd532a4f39aa952bd88e021aa9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:06:23.386595 11573 raft_consensus.cc:2258] T a5d6d6c961504f3aba94819ef4a13123 P caa4aecd532a4f39aa952bd88e021aa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:23.386562 10209 catalog_manager.cc:4841] TS acd3c54a49cd4a6ebe2cbd2a34ea80af (127.3.214.194:36209): tablet a5d6d6c961504f3aba94819ef4a13123 (table test-table [id=cf71fe47095a4808bbc43e601fa5ef4b]) successfully deleted
I20260924 16:06:23.389014 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 10191
I20260924 16:06:23.390403 10253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:23.391870 11573 ts_tablet_manager.cc:1905] T a5d6d6c961504f3aba94819ef4a13123 P caa4aecd532a4f39aa952bd88e021aa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:23.397653 11572 ts_tablet_manager.cc:1918] T a5d6d6c961504f3aba94819ef4a13123 P 91edf6392c294ba1abeb6a0ffca2a21f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:06:23.398171 11572 log.cc:1196] T a5d6d6c961504f3aba94819ef4a13123 P 91edf6392c294ba1abeb6a0ffca2a21f: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/a5d6d6c961504f3aba94819ef4a13123
I20260924 16:06:23.398886 11572 ts_tablet_manager.cc:1939] T a5d6d6c961504f3aba94819ef4a13123 P 91edf6392c294ba1abeb6a0ffca2a21f: Deleting consensus metadata
I20260924 16:06:23.400136 11573 ts_tablet_manager.cc:1918] T a5d6d6c961504f3aba94819ef4a13123 P caa4aecd532a4f39aa952bd88e021aa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:06:23.400535 11573 log.cc:1196] T a5d6d6c961504f3aba94819ef4a13123 P caa4aecd532a4f39aa952bd88e021aa9: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/wal/wals/a5d6d6c961504f3aba94819ef4a13123
I20260924 16:06:23.401274 11573 ts_tablet_manager.cc:1939] T a5d6d6c961504f3aba94819ef4a13123 P caa4aecd532a4f39aa952bd88e021aa9: Deleting consensus metadata
I20260924 16:06:23.414474 11524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:23.371755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42474 (local address 127.3.214.253:37401)
0924 16:06:23.371949 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:23.371976 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:23.387762 (+ 15786us) server_negotiation.cc:407] Connection header received
0924 16:06:23.388110 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:23.388117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:23.388213 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:23.388289 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:23.391012 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.392199 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.400938 (+ 8739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.401709 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.401836 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:23.406320 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:23.406331 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:23.406336 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:23.406402 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:23.410649 (+ 4247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:23.411869 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:23.411877 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:23.411881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:23.412089 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:23.412558 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:23.412563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:23.412567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:23.413001 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:23.413305 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:23.413702 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 16:06:23.414015 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:06:23.434310 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:23.388057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.3.214.252:35111)
0924 16:06:23.388359 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:23.388366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:23.414699 (+ 26333us) server_negotiation.cc:407] Connection header received
0924 16:06:23.415434 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:23.415441 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:23.415546 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:23.415638 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:23.417716 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.418912 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.420934 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.421606 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.421773 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:23.425935 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:23.425945 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:23.425951 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:23.426017 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:23.429986 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:23.431145 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:23.431151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:23.431156 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:23.431378 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:23.431837 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:23.431842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:23.431845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:23.432312 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:23.432589 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:23.433357 (+ 768us) server_negotiation.cc:299] Negotiation successful
0924 16:06:23.433648 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260924 16:06:23.550966 10208 catalog_manager.cc:4841] TS 91edf6392c294ba1abeb6a0ffca2a21f (127.3.214.195:37151): tablet a5d6d6c961504f3aba94819ef4a13123 (table test-table [id=cf71fe47095a4808bbc43e601fa5ef4b]) successfully deleted
I20260924 16:06:23.550999 10209 catalog_manager.cc:4841] TS caa4aecd532a4f39aa952bd88e021aa9 (127.3.214.193:37329): tablet a5d6d6c961504f3aba94819ef4a13123 (table test-table [id=cf71fe47095a4808bbc43e601fa5ef4b]) successfully deleted
I20260924 16:06:23.553196 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 10191
I20260924 16:06:23.605422 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:23.584367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43274 (local address 127.3.214.252:35111)
0924 16:06:23.584621 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:23.584628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:23.586094 (+ 1466us) server_negotiation.cc:407] Connection header received
0924 16:06:23.586512 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:23.586518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:23.586612 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:23.586692 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:23.589303 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.590441 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.592715 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.593421 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.593527 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:23.597973 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:23.597982 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:23.597988 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:23.598072 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:23.602094 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:23.602912 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:23.602919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:23.602923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:23.603062 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:23.603557 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:23.603565 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:23.603568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:23.604043 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:23.604332 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:23.604740 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 16:06:23.605040 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260924 16:06:23.613234 11524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:23.584811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42476 (local address 127.3.214.253:37401)
0924 16:06:23.585004 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:23.585010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:23.595390 (+ 10380us) server_negotiation.cc:407] Connection header received
0924 16:06:23.595689 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:23.595699 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:23.595836 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:23.595981 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:23.598962 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.600244 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.602258 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:23.602963 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:23.603131 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:23.606401 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:23.606411 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:23.606416 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:23.606462 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:23.609816 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:23.610877 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:23.610884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:23.610888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:23.611068 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:23.611507 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:23.611512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:23.611515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:23.611922 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:23.612170 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:23.612600 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 16:06:23.612882 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 16:06:24.345255 11583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:24.344452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.194:33633)
0924 16:06:24.344928 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:24.345072 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":243,"threads_started":1}
W20260924 16:06:24.345775 11364 heartbeater.cc:644] Failed to heartbeat to 127.3.214.254:43105 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
I20260924 16:06:24.351617 11584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:24.348630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.195:48219)
0924 16:06:24.351265 (+ 2635us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:24.351419 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2494,"thread_start_us":2370,"threads_started":1}
I20260924 16:06:24.351661 11585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:24.350473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.193:36141)
0924 16:06:24.351282 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:24.351444 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":485,"threads_started":1}
W20260924 16:06:24.352141 11509 heartbeater.cc:644] Failed to heartbeat to 127.3.214.254:43105 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
W20260924 16:06:24.352190 11215 heartbeater.cc:644] Failed to heartbeat to 127.3.214.254:43105 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
I20260924 16:06:24.863938 11586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 608e51b0fef1488ba73b3f205c4758d5)
I20260924 16:06:24.864250 11586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: false } }
I20260924 16:06:24.865161 11587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 608e51b0fef1488ba73b3f205c4758d5)
I20260924 16:06:24.865399 11587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: false } }
I20260924 16:06:24.865810 11586 leader_election.cc:288] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e51b0fef1488ba73b3f205c4758d5 (127.3.214.254:43105), 5849dfa252f444578e275b75c9d9e686 (127.3.214.253:37401)
I20260924 16:06:24.866467 11587 leader_election.cc:288] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e51b0fef1488ba73b3f205c4758d5 (127.3.214.254:43105), 6fa32a0f89eb4e7ab6846d123a8f175c (127.3.214.252:35111)
I20260924 16:06:24.876991 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:24.866676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:60632)
0924 16:06:24.876391 (+ 9715us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:24.876688 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9510,"threads_started":1}
I20260924 16:06:24.877475 11591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:24.866041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:60630)
0924 16:06:24.876763 (+ 10722us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:24.877091 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10554,"threads_started":1}
W20260924 16:06:24.878996 10819 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
W20260924 16:06:24.880194 11016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
I20260924 16:06:24.887917 11591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:24.866141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:37401 (local address 127.0.0.1:38010)
0924 16:06:24.877884 (+ 11743us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:24.877908 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:24.878174 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:24.879476 (+ 1302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:24.879482 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:24.879513 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:24.879869 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:24.879883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:24.882217 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:24.882225 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:24.884697 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:24.884717 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:24.886792 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:24.886799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:24.887047 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:24.887185 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 16:06:24.887278 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 16:06:24.887528 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11641,"thread_start_us":13080,"threads_started":1}
I20260924 16:06:24.888972 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:24.888366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:60642)
0924 16:06:24.888574 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:24.888775 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 16:06:24.889312 11016 leader_election.cc:334] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 608e51b0fef1488ba73b3f205c4758d5 (127.3.214.254:43105): Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
I20260924 16:06:24.889724 11589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:24.866288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38010 (local address 127.3.214.253:37401)
0924 16:06:24.873625 (+ 7337us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:24.873633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:24.879157 (+ 5524us) server_negotiation.cc:407] Connection header received
0924 16:06:24.879228 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:24.879234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:24.879308 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:24.880084 (+ 776us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:24.880348 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:24.881983 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:24.884945 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:24.886558 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:24.888742 (+ 2184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:24.888857 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:24.888925 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 16:06:24.889212 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":7166,"thread_start_us":6885,"threads_started":1}
I20260924 16:06:24.890161 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:24.873985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:35111 (local address 127.0.0.1:34606)
0924 16:06:24.878849 (+ 4864us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:24.878870 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:24.879018 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:24.880620 (+ 1602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:24.880633 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:24.880679 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:24.881270 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:24.881284 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:24.883339 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:24.883346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:24.885861 (+ 2515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:24.885885 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:24.888938 (+ 3053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:24.888946 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:24.889200 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:24.889245 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 16:06:24.889361 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 16:06:24.889565 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4691,"mutex_wait_us":5,"thread_start_us":1971,"threads_started":1}
I20260924 16:06:24.890228 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:24.874078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34606 (local address 127.3.214.252:35111)
0924 16:06:24.879950 (+ 5872us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:24.879958 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:24.879991 (+ 33us) server_negotiation.cc:407] Connection header received
0924 16:06:24.880084 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:24.880091 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:24.880193 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:24.880353 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:24.881599 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:24.883123 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:24.886141 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:24.888352 (+ 2211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:24.888539 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:24.888709 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:24.889706 (+ 997us) server_negotiation.cc:299] Negotiation successful
0924 16:06:24.889877 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5633,"thread_start_us":5440,"threads_started":1}
I20260924 16:06:24.890820 10846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5849dfa252f444578e275b75c9d9e686" is_pre_election: true
I20260924 16:06:24.891342 10846 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fa32a0f89eb4e7ab6846d123a8f175c in term 1.
I20260924 16:06:24.891655 11042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5849dfa252f444578e275b75c9d9e686" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" is_pre_election: true
I20260924 16:06:24.892151 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:24.891645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:60654)
0924 16:06:24.891848 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:24.891983 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 16:06:24.892113 11042 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5849dfa252f444578e275b75c9d9e686 in term 1.
I20260924 16:06:24.892022 11018 leader_election.cc:302] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5849dfa252f444578e275b75c9d9e686, 6fa32a0f89eb4e7ab6846d123a8f175c; no voters: 608e51b0fef1488ba73b3f205c4758d5
W20260924 16:06:24.892445 10819 leader_election.cc:334] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 608e51b0fef1488ba73b3f205c4758d5 (127.3.214.254:43105): Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
I20260924 16:06:24.892539 11586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 16:06:24.892664 11586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 608e51b0fef1488ba73b3f205c4758d5)
I20260924 16:06:24.892776 11586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:06:24.892810 10820 leader_election.cc:302] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5849dfa252f444578e275b75c9d9e686, 6fa32a0f89eb4e7ab6846d123a8f175c; no voters: 608e51b0fef1488ba73b3f205c4758d5
I20260924 16:06:24.893162 11587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 16:06:24.893276 11587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 608e51b0fef1488ba73b3f205c4758d5)
I20260924 16:06:24.893370 11587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:06:24.898319 11586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: false } }
I20260924 16:06:24.898314 11587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: false } }
I20260924 16:06:24.899251 11587 leader_election.cc:288] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [CANDIDATE]: Term 2 election: Requested vote from peers 608e51b0fef1488ba73b3f205c4758d5 (127.3.214.254:43105), 6fa32a0f89eb4e7ab6846d123a8f175c (127.3.214.252:35111)
I20260924 16:06:24.899266 11586 leader_election.cc:288] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [CANDIDATE]: Term 2 election: Requested vote from peers 608e51b0fef1488ba73b3f205c4758d5 (127.3.214.254:43105), 5849dfa252f444578e275b75c9d9e686 (127.3.214.253:37401)
I20260924 16:06:24.900033 10846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5849dfa252f444578e275b75c9d9e686"
I20260924 16:06:24.899993 11042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5849dfa252f444578e275b75c9d9e686" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c"
I20260924 16:06:24.900132 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:24.899479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:60658)
0924 16:06:24.899764 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:24.899884 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260924 16:06:24.900367 10846 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6fa32a0f89eb4e7ab6846d123a8f175c in current term 2: Already voted for candidate 5849dfa252f444578e275b75c9d9e686 in this term.
I20260924 16:06:24.900364 11042 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5849dfa252f444578e275b75c9d9e686 in current term 2: Already voted for candidate 6fa32a0f89eb4e7ab6846d123a8f175c in this term.
I20260924 16:06:24.900887 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:24.899691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:60662)
0924 16:06:24.900622 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:24.900721 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":807}
I20260924 16:06:24.901679 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:24.901229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:60678)
0924 16:06:24.901375 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:24.901496 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 16:06:24.902064 10819 leader_election.cc:334] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 608e51b0fef1488ba73b3f205c4758d5 (127.3.214.254:43105): Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
I20260924 16:06:24.902273 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:24.901688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:60684)
0924 16:06:24.901957 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:24.902065 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 16:06:24.902338 10819 leader_election.cc:302] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5849dfa252f444578e275b75c9d9e686; no voters: 608e51b0fef1488ba73b3f205c4758d5, 6fa32a0f89eb4e7ab6846d123a8f175c
W20260924 16:06:24.902493 11016 leader_election.cc:334] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 608e51b0fef1488ba73b3f205c4758d5 (127.3.214.254:43105): Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
I20260924 16:06:24.902637 11016 leader_election.cc:302] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fa32a0f89eb4e7ab6846d123a8f175c; no voters: 5849dfa252f444578e275b75c9d9e686, 608e51b0fef1488ba73b3f205c4758d5
I20260924 16:06:24.902789 11587 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 16:06:24.902976 11586 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 16:06:25.347464 11596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:25.346622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.194:38667)
0924 16:06:25.347146 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:25.347251 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":276,"threads_started":1}
I20260924 16:06:25.354094 11598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:25.353120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.195:60029)
0924 16:06:25.353766 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:25.353908 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":327,"threads_started":1}
I20260924 16:06:25.354385 11597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:25.352977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.193:56289)
0924 16:06:25.353961 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:25.354069 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":855,"thread_start_us":737,"threads_started":1}
I20260924 16:06:26.349366 11599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.348426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.194:55281)
0924 16:06:26.349033 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.349150 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":328,"threads_started":1}
I20260924 16:06:26.355808 11600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.355039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.195:55671)
0924 16:06:26.355528 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.355621 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":259,"threads_started":1}
I20260924 16:06:26.356227 11601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.355365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.193:44847)
0924 16:06:26.355878 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.355999 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":273,"threads_started":1}
I20260924 16:06:26.472316 11602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:06:26.472711 11602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: false } }
I20260924 16:06:26.473661 11602 leader_election.cc:288] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e51b0fef1488ba73b3f205c4758d5 (127.3.214.254:43105), 5849dfa252f444578e275b75c9d9e686 (127.3.214.253:37401)
I20260924 16:06:26.474725 10846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5849dfa252f444578e275b75c9d9e686" is_pre_election: true
I20260924 16:06:26.475047 11603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.473930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:60938)
0924 16:06:26.474644 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.474807 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":375,"threads_started":1}
I20260924 16:06:26.475119 10846 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fa32a0f89eb4e7ab6846d123a8f175c in term 2.
I20260924 16:06:26.475666 11018 leader_election.cc:302] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5849dfa252f444578e275b75c9d9e686, 6fa32a0f89eb4e7ab6846d123a8f175c; no voters:
I20260924 16:06:26.476011 11602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 16:06:26.476181 11602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:06:26.476316 11602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 2 FOLLOWER]: Advancing to term 3
I20260924 16:06:26.477038 11603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.476505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:60942)
0924 16:06:26.476697 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.476824 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 16:06:26.477329 11016 leader_election.cc:334] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 608e51b0fef1488ba73b3f205c4758d5 (127.3.214.254:43105): Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
I20260924 16:06:26.479241 11602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: false } }
I20260924 16:06:26.480114 11602 leader_election.cc:288] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [CANDIDATE]: Term 3 election: Requested vote from peers 608e51b0fef1488ba73b3f205c4758d5 (127.3.214.254:43105), 5849dfa252f444578e275b75c9d9e686 (127.3.214.253:37401)
I20260924 16:06:26.480788 11603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.480254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:60958)
0924 16:06:26.480455 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.480593 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 16:06:26.481009 10846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5849dfa252f444578e275b75c9d9e686"
I20260924 16:06:26.481318 10846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 2 FOLLOWER]: Advancing to term 3
I20260924 16:06:26.481875 11603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.481553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:60974)
0924 16:06:26.481683 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.481759 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 16:06:26.482062 11016 leader_election.cc:334] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 608e51b0fef1488ba73b3f205c4758d5 (127.3.214.254:43105): Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
I20260924 16:06:26.483826 10846 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa32a0f89eb4e7ab6846d123a8f175c in term 3.
I20260924 16:06:26.484335 11018 leader_election.cc:302] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5849dfa252f444578e275b75c9d9e686, 6fa32a0f89eb4e7ab6846d123a8f175c; no voters: 608e51b0fef1488ba73b3f205c4758d5
I20260924 16:06:26.484721 11602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 3 FOLLOWER]: Leader election won for term 3
I20260924 16:06:26.485298 11602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [term 3 LEADER]: Becoming Leader. State: Replica: 6fa32a0f89eb4e7ab6846d123a8f175c, State: Running, Role: LEADER
I20260924 16:06:26.485810 11602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: false } }
I20260924 16:06:26.487862 11605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fa32a0f89eb4e7ab6846d123a8f175c. Latest consensus state: current_term: 3 leader_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: false } } }
I20260924 16:06:26.488258 11605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:26.489136 11607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:26.493840 11607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:26.494951 11607 catalog_manager.cc:1259] Loaded cluster ID: 1d01b2f7e31946b0b6b5616bf5533586
I20260924 16:06:26.495055 11607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:26.496583 11607 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:26.497552 11607 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c: Loaded TSK: 0
I20260924 16:06:26.498854 11607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:06:26.526878 11032 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:06:26.534520 11032 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:06:26.540096 11603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.539429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:60992)
0924 16:06:26.539753 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.539885 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260924 16:06:26.540107 10846 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [term 3 FOLLOWER]: Refusing update from remote peer 6fa32a0f89eb4e7ab6846d123a8f175c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 16:06:26.541006 11605 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 16:06:26.541399 11603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.540959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:60996)
0924 16:06:26.541143 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.541254 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 16:06:26.541675 11016 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c -> Peer 608e51b0fef1488ba73b3f205c4758d5 (127.3.214.254:43105): Couldn't send request to peer 608e51b0fef1488ba73b3f205c4758d5. Status: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 16:06:26.545210 11609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fa32a0f89eb4e7ab6846d123a8f175c. Latest consensus state: current_term: 3 leader_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: false } } }
I20260924 16:06:26.545604 11609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:06:26.546764 11605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: false } } }
I20260924 16:06:26.547257 11605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:26.547603 11609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "608e51b0fef1488ba73b3f205c4758d5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 43105 } } peers { permanent_uuid: "5849dfa252f444578e275b75c9d9e686" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 37401 } attrs { promote: false } } peers { permanent_uuid: "6fa32a0f89eb4e7ab6846d123a8f175c" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 35111 } attrs { promote: false } } }
I20260924 16:06:26.547884 11609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5849dfa252f444578e275b75c9d9e686 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:06:26.566987 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.557669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:36209 (local address 127.0.0.1:52448)
0924 16:06:26.558567 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.558588 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:26.558725 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:26.559764 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:26.559770 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:26.559811 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:26.560171 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:26.560184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.561413 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.561417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:26.563673 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:26.563693 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.565704 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.565711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:26.565981 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:26.566053 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 16:06:26.566195 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 16:06:26.566391 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":3,"thread_start_us":605,"threads_started":1}
I20260924 16:06:26.567623 11616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.557892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.3.214.194:36209)
0924 16:06:26.559310 (+ 1418us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:26.559317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:26.559373 (+ 56us) server_negotiation.cc:407] Connection header received
0924 16:06:26.559477 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:26.559482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:26.559577 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:26.559695 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:26.560364 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.561272 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.563884 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.564915 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.566904 (+ 1989us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:26.567035 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:26.567097 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 16:06:26.567265 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1251,"thread_start_us":1089,"threads_started":1}
I20260924 16:06:26.568794 11297 tablet_service.cc:1467] Processing CreateTablet for tablet 1eb3bc371dcf4bc985ce6ea040c74038 (DEFAULT_TABLE table=test-table [id=f608ddc5208a4ee79a4697e318830ab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:26.569382 11297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eb3bc371dcf4bc985ce6ea040c74038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:26.573285 11617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.558478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:37151 (local address 127.0.0.1:39826)
0924 16:06:26.559566 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.559583 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:26.559708 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:26.565589 (+ 5881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:26.565595 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:26.565667 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:26.567049 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:26.567063 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.568670 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.568676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:26.570610 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:26.570629 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.572351 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.572358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:26.572567 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:26.572609 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 16:06:26.572678 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 16:06:26.572902 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":954,"mutex_wait_us":644,"thread_start_us":775,"threads_started":1}
I20260924 16:06:26.573961 11603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.557246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:37329 (local address 127.0.0.1:37664)
0924 16:06:26.557739 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.557768 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:26.557917 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:26.566593 (+ 8676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:26.566597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:26.566658 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:26.567118 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:26.567127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.568866 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.568872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:26.571083 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:26.571100 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.572863 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.572871 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:26.573063 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:26.573103 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 16:06:26.573210 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 16:06:26.573379 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":2}
I20260924 16:06:26.574361 11618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.558902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39826 (local address 127.3.214.195:37151)
0924 16:06:26.565105 (+ 6203us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:26.565112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:26.565134 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:06:26.565246 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:26.565254 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:26.565361 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:26.565479 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:26.567311 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.568499 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.570876 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.572102 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.573658 (+ 1556us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:26.573780 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:26.573837 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 16:06:26.574000 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6043,"thread_start_us":5909,"threads_started":1}
I20260924 16:06:26.574360 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.557415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37664 (local address 127.3.214.193:37329)
0924 16:06:26.566217 (+ 8802us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:26.566224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:26.566245 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:06:26.566342 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:26.566347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:26.566428 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:26.566535 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:26.567886 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.568703 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.571286 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.572565 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.572721 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:26.572836 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:26.573270 (+ 434us) server_negotiation.cc:299] Negotiation successful
0924 16:06:26.573824 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8597,"thread_start_us":8668,"threads_started":1}
I20260924 16:06:26.576982 11619 tablet_bootstrap.cc:492] T 1eb3bc371dcf4bc985ce6ea040c74038 P acd3c54a49cd4a6ebe2cbd2a34ea80af: Bootstrap starting.
I20260924 16:06:26.577312 11441 tablet_service.cc:1467] Processing CreateTablet for tablet 1eb3bc371dcf4bc985ce6ea040c74038 (DEFAULT_TABLE table=test-table [id=f608ddc5208a4ee79a4697e318830ab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:26.577525 11149 tablet_service.cc:1467] Processing CreateTablet for tablet 1eb3bc371dcf4bc985ce6ea040c74038 (DEFAULT_TABLE table=test-table [id=f608ddc5208a4ee79a4697e318830ab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:26.577885 11441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eb3bc371dcf4bc985ce6ea040c74038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:26.577991 11149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eb3bc371dcf4bc985ce6ea040c74038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:26.585485 11619 tablet_bootstrap.cc:654] T 1eb3bc371dcf4bc985ce6ea040c74038 P acd3c54a49cd4a6ebe2cbd2a34ea80af: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:26.586210 11621 tablet_bootstrap.cc:492] T 1eb3bc371dcf4bc985ce6ea040c74038 P 91edf6392c294ba1abeb6a0ffca2a21f: Bootstrap starting.
I20260924 16:06:26.587211 11620 tablet_bootstrap.cc:492] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9: Bootstrap starting.
I20260924 16:06:26.588313 11621 tablet_bootstrap.cc:654] T 1eb3bc371dcf4bc985ce6ea040c74038 P 91edf6392c294ba1abeb6a0ffca2a21f: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:26.588497 11619 tablet_bootstrap.cc:492] T 1eb3bc371dcf4bc985ce6ea040c74038 P acd3c54a49cd4a6ebe2cbd2a34ea80af: No bootstrap required, opened a new log
I20260924 16:06:26.588717 11619 ts_tablet_manager.cc:1397] T 1eb3bc371dcf4bc985ce6ea040c74038 P acd3c54a49cd4a6ebe2cbd2a34ea80af: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260924 16:06:26.589232 11620 tablet_bootstrap.cc:654] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:26.589624 11619 raft_consensus.cc:348] T 1eb3bc371dcf4bc985ce6ea040c74038 P acd3c54a49cd4a6ebe2cbd2a34ea80af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa4aecd532a4f39aa952bd88e021aa9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37329 } } peers { permanent_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 36209 } } peers { permanent_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37151 } }
I20260924 16:06:26.589960 11619 raft_consensus.cc:374] T 1eb3bc371dcf4bc985ce6ea040c74038 P acd3c54a49cd4a6ebe2cbd2a34ea80af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:26.590135 11619 raft_consensus.cc:729] T 1eb3bc371dcf4bc985ce6ea040c74038 P acd3c54a49cd4a6ebe2cbd2a34ea80af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd3c54a49cd4a6ebe2cbd2a34ea80af, State: Initialized, Role: FOLLOWER
I20260924 16:06:26.590476 11619 consensus_queue.cc:260] T 1eb3bc371dcf4bc985ce6ea040c74038 P acd3c54a49cd4a6ebe2cbd2a34ea80af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa4aecd532a4f39aa952bd88e021aa9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37329 } } peers { permanent_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 36209 } } peers { permanent_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37151 } }
I20260924 16:06:26.593220 11621 tablet_bootstrap.cc:492] T 1eb3bc371dcf4bc985ce6ea040c74038 P 91edf6392c294ba1abeb6a0ffca2a21f: No bootstrap required, opened a new log
I20260924 16:06:26.593503 11621 ts_tablet_manager.cc:1397] T 1eb3bc371dcf4bc985ce6ea040c74038 P 91edf6392c294ba1abeb6a0ffca2a21f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 16:06:26.594579 11621 raft_consensus.cc:348] T 1eb3bc371dcf4bc985ce6ea040c74038 P 91edf6392c294ba1abeb6a0ffca2a21f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa4aecd532a4f39aa952bd88e021aa9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37329 } } peers { permanent_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 36209 } } peers { permanent_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37151 } }
I20260924 16:06:26.594990 11621 raft_consensus.cc:374] T 1eb3bc371dcf4bc985ce6ea040c74038 P 91edf6392c294ba1abeb6a0ffca2a21f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:26.595132 11621 raft_consensus.cc:729] T 1eb3bc371dcf4bc985ce6ea040c74038 P 91edf6392c294ba1abeb6a0ffca2a21f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91edf6392c294ba1abeb6a0ffca2a21f, State: Initialized, Role: FOLLOWER
I20260924 16:06:26.595504 11621 consensus_queue.cc:260] T 1eb3bc371dcf4bc985ce6ea040c74038 P 91edf6392c294ba1abeb6a0ffca2a21f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa4aecd532a4f39aa952bd88e021aa9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37329 } } peers { permanent_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 36209 } } peers { permanent_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37151 } }
I20260924 16:06:26.597213 11619 ts_tablet_manager.cc:1428] T 1eb3bc371dcf4bc985ce6ea040c74038 P acd3c54a49cd4a6ebe2cbd2a34ea80af: Time spent starting tablet: real 0.008s user 0.000s sys 0.007s
I20260924 16:06:26.598621 11599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.598013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.194:51719)
0924 16:06:26.598329 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.598434 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260924 16:06:26.600512 11363 heartbeater.cc:503] Master 127.3.214.252:35111 was elected leader, sending a full tablet report...
I20260924 16:06:26.600884 11620 tablet_bootstrap.cc:492] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9: No bootstrap required, opened a new log
I20260924 16:06:26.601123 11620 ts_tablet_manager.cc:1397] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260924 16:06:26.602252 11620 raft_consensus.cc:348] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa4aecd532a4f39aa952bd88e021aa9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37329 } } peers { permanent_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 36209 } } peers { permanent_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37151 } }
I20260924 16:06:26.602880 11620 raft_consensus.cc:374] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:26.603009 11620 raft_consensus.cc:729] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caa4aecd532a4f39aa952bd88e021aa9, State: Initialized, Role: FOLLOWER
I20260924 16:06:26.603322 11620 consensus_queue.cc:260] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa4aecd532a4f39aa952bd88e021aa9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37329 } } peers { permanent_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 36209 } } peers { permanent_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37151 } }
I20260924 16:06:26.608173 11621 ts_tablet_manager.cc:1428] T 1eb3bc371dcf4bc985ce6ea040c74038 P 91edf6392c294ba1abeb6a0ffca2a21f: Time spent starting tablet: real 0.014s user 0.005s sys 0.003s
I20260924 16:06:26.610814 11600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.610273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.195:44135)
0924 16:06:26.610517 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.610624 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260924 16:06:26.611688 11599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.610976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.194:60533)
0924 16:06:26.611337 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.611454 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216}
I20260924 16:06:26.611893 11508 heartbeater.cc:503] Master 127.3.214.252:35111 was elected leader, sending a full tablet report...
I20260924 16:06:26.613544 11600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.613028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.195:57719)
0924 16:06:26.613260 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.613366 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260924 16:06:26.616852 11620 ts_tablet_manager.cc:1428] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9: Time spent starting tablet: real 0.015s user 0.008s sys 0.004s
I20260924 16:06:26.618363 11601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.617691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.193:59281)
0924 16:06:26.617982 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.618129 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260924 16:06:26.619688 11601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.619212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.193:45677)
0924 16:06:26.619435 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.619521 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260924 16:06:26.620549 11214 heartbeater.cc:503] Master 127.3.214.252:35111 was elected leader, sending a full tablet report...
I20260924 16:06:26.824968 11627 raft_consensus.cc:482] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:06:26.825224 11627 raft_consensus.cc:504] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa4aecd532a4f39aa952bd88e021aa9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37329 } } peers { permanent_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 36209 } } peers { permanent_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37151 } }
I20260924 16:06:26.827128 11627 leader_election.cc:288] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acd3c54a49cd4a6ebe2cbd2a34ea80af (127.3.214.194:36209), 91edf6392c294ba1abeb6a0ffca2a21f (127.3.214.195:37151)
I20260924 16:06:26.836061 11616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.827584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:52965 (local address 127.3.214.194:36209)
0924 16:06:26.827916 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:26.827924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:26.827958 (+ 34us) server_negotiation.cc:407] Connection header received
0924 16:06:26.828136 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:26.828141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:26.828242 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:26.828370 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:26.829103 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.830254 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.833194 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.834650 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.834851 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:26.834942 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:26.835578 (+ 636us) server_negotiation.cc:299] Negotiation successful
0924 16:06:26.835770 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260924 16:06:26.836056 11618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.827718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:38897 (local address 127.3.214.195:37151)
0924 16:06:26.827927 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:26.827932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:26.828558 (+ 626us) server_negotiation.cc:407] Connection header received
0924 16:06:26.828738 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:26.828742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:26.828814 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:26.828898 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:26.829426 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.830527 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.833202 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.834646 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.834851 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:26.834942 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:26.835577 (+ 635us) server_negotiation.cc:299] Negotiation successful
0924 16:06:26.835749 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 16:06:26.836189 11601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.827407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:36209 (local address 127.3.214.193:52965)
0924 16:06:26.827749 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.827770 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:26.827949 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:26.828426 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:26.828432 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:26.828479 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:26.828912 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:26.828931 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.830438 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.830443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:26.832933 (+ 2490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:26.832958 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.835004 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.835012 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:26.835382 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:26.835410 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 16:06:26.835521 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 16:06:26.835728 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":149}
I20260924 16:06:26.836230 11628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.827599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:37151 (local address 127.3.214.193:38897)
0924 16:06:26.828417 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.828436 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:26.828568 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:26.828996 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:26.829001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:26.829030 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:26.829280 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:26.829288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.830668 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.830672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:26.832933 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:26.832960 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.835007 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.835012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:26.835337 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:26.835383 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 16:06:26.835521 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 16:06:26.835728 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":114,"thread_start_us":450,"threads_started":1}
I20260924 16:06:26.837209 11461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb3bc371dcf4bc985ce6ea040c74038" candidate_uuid: "caa4aecd532a4f39aa952bd88e021aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" is_pre_election: true
I20260924 16:06:26.837563 11461 raft_consensus.cc:2454] T 1eb3bc371dcf4bc985ce6ea040c74038 P 91edf6392c294ba1abeb6a0ffca2a21f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caa4aecd532a4f39aa952bd88e021aa9 in term 0.
I20260924 16:06:26.837438 11317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb3bc371dcf4bc985ce6ea040c74038" candidate_uuid: "caa4aecd532a4f39aa952bd88e021aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" is_pre_election: true
I20260924 16:06:26.837908 11317 raft_consensus.cc:2454] T 1eb3bc371dcf4bc985ce6ea040c74038 P acd3c54a49cd4a6ebe2cbd2a34ea80af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caa4aecd532a4f39aa952bd88e021aa9 in term 0.
I20260924 16:06:26.838254 11102 leader_election.cc:302] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91edf6392c294ba1abeb6a0ffca2a21f, caa4aecd532a4f39aa952bd88e021aa9; no voters:
I20260924 16:06:26.838835 11627 raft_consensus.cc:2781] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:06:26.838982 11627 raft_consensus.cc:482] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:06:26.839090 11627 raft_consensus.cc:3037] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:26.841523 11627 raft_consensus.cc:504] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa4aecd532a4f39aa952bd88e021aa9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37329 } } peers { permanent_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 36209 } } peers { permanent_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37151 } }
I20260924 16:06:26.842238 11627 leader_election.cc:288] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9 [CANDIDATE]: Term 1 election: Requested vote from peers acd3c54a49cd4a6ebe2cbd2a34ea80af (127.3.214.194:36209), 91edf6392c294ba1abeb6a0ffca2a21f (127.3.214.195:37151)
I20260924 16:06:26.842826 11461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb3bc371dcf4bc985ce6ea040c74038" candidate_uuid: "caa4aecd532a4f39aa952bd88e021aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91edf6392c294ba1abeb6a0ffca2a21f"
I20260924 16:06:26.842850 11317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb3bc371dcf4bc985ce6ea040c74038" candidate_uuid: "caa4aecd532a4f39aa952bd88e021aa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af"
I20260924 16:06:26.843037 11461 raft_consensus.cc:3037] T 1eb3bc371dcf4bc985ce6ea040c74038 P 91edf6392c294ba1abeb6a0ffca2a21f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:26.843081 11317 raft_consensus.cc:3037] T 1eb3bc371dcf4bc985ce6ea040c74038 P acd3c54a49cd4a6ebe2cbd2a34ea80af [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:26.845424 11461 raft_consensus.cc:2454] T 1eb3bc371dcf4bc985ce6ea040c74038 P 91edf6392c294ba1abeb6a0ffca2a21f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caa4aecd532a4f39aa952bd88e021aa9 in term 1.
I20260924 16:06:26.845626 11317 raft_consensus.cc:2454] T 1eb3bc371dcf4bc985ce6ea040c74038 P acd3c54a49cd4a6ebe2cbd2a34ea80af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caa4aecd532a4f39aa952bd88e021aa9 in term 1.
I20260924 16:06:26.845969 11102 leader_election.cc:302] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91edf6392c294ba1abeb6a0ffca2a21f, caa4aecd532a4f39aa952bd88e021aa9; no voters:
I20260924 16:06:26.846410 11627 raft_consensus.cc:2781] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:26.846968 11627 raft_consensus.cc:686] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9 [term 1 LEADER]: Becoming Leader. State: Replica: caa4aecd532a4f39aa952bd88e021aa9, State: Running, Role: LEADER
I20260924 16:06:26.847492 11627 consensus_queue.cc:237] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa4aecd532a4f39aa952bd88e021aa9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37329 } } peers { permanent_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 36209 } } peers { permanent_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37151 } }
I20260924 16:06:26.848399 11601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.847814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.193:51887)
0924 16:06:26.848075 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.848183 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 16:06:26.852017 11032 catalog_manager.cc:5495] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9 reported cstate change: term changed from 0 to 1, leader changed from <none> to caa4aecd532a4f39aa952bd88e021aa9 (127.3.214.193). New cstate: current_term: 1 leader_uuid: "caa4aecd532a4f39aa952bd88e021aa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa4aecd532a4f39aa952bd88e021aa9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 36209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37151 } health_report { overall_health: UNKNOWN } } }
I20260924 16:06:26.879180 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.873365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37678 (local address 127.3.214.193:37329)
0924 16:06:26.873643 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:26.873650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:26.873701 (+ 51us) server_negotiation.cc:407] Connection header received
0924 16:06:26.873814 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:26.873818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:26.873905 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:26.874075 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:26.874513 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.875229 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.876752 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.877220 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.877340 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:26.877797 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:26.878505 (+ 708us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:26.878648 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:26.878766 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 16:06:26.878863 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260924 16:06:26.884758 11461 raft_consensus.cc:1261] T 1eb3bc371dcf4bc985ce6ea040c74038 P 91edf6392c294ba1abeb6a0ffca2a21f [term 1 FOLLOWER]: Refusing update from remote peer caa4aecd532a4f39aa952bd88e021aa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:26.885372 11317 raft_consensus.cc:1261] T 1eb3bc371dcf4bc985ce6ea040c74038 P acd3c54a49cd4a6ebe2cbd2a34ea80af [term 1 FOLLOWER]: Refusing update from remote peer caa4aecd532a4f39aa952bd88e021aa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:26.885643 11629 consensus_queue.cc:1035] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91edf6392c294ba1abeb6a0ffca2a21f" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:26.886391 11627 consensus_queue.cc:1035] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acd3c54a49cd4a6ebe2cbd2a34ea80af" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 36209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:26.888827 11600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.888232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.195:50277)
0924 16:06:26.888499 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.888611 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260924 16:06:26.889616 11599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.888974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.194:55735)
0924 16:06:26.889250 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.889378 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260924 16:06:26.896080 11632 mvcc.cc:205] Tried to move back new op lower bound from 7332929482270105600 to 7332929482132103168. Current Snapshot: MvccSnapshot[applied={T|T < 7332929482270105600}]
I20260924 16:06:26.898237 11600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.897490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.195:53495)
0924 16:06:26.897821 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.897954 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
I20260924 16:06:26.900022 11601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.899322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.193:52481)
0924 16:06:26.899691 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.899800 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229}
I20260924 16:06:26.904034 11599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.903423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.194:39819)
0924 16:06:26.903704 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.903812 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260924 16:06:26.904098 11601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.903431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.3.214.193:34935)
0924 16:06:26.903705 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:26.903865 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260924 16:06:26.909379 11634 mvcc.cc:205] Tried to move back new op lower bound from 7332929482270105600 to 7332929482132103168. Current Snapshot: MvccSnapshot[applied={T|T < 7332929482270105600}]
I20260924 16:06:26.923574 11033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:26.923980 11033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:26.931707 11033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fa32a0f89eb4e7ab6846d123a8f175c: Sending DeleteTablet for 3 replicas of tablet 1eb3bc371dcf4bc985ce6ea040c74038
I20260924 16:06:26.933791 11441 tablet_service.cc:1514] Processing DeleteTablet for tablet 1eb3bc371dcf4bc985ce6ea040c74038 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:26 UTC) from {username='oryx', principal='oryx/127.3.214.252@KRBTEST.COM'} at 127.0.0.1:39826
I20260924 16:06:26.934327 11297 tablet_service.cc:1514] Processing DeleteTablet for tablet 1eb3bc371dcf4bc985ce6ea040c74038 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:26 UTC) from {username='oryx', principal='oryx/127.3.214.252@KRBTEST.COM'} at 127.0.0.1:52448
I20260924 16:06:26.934515 11149 tablet_service.cc:1514] Processing DeleteTablet for tablet 1eb3bc371dcf4bc985ce6ea040c74038 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:26 UTC) from {username='oryx', principal='oryx/127.3.214.252@KRBTEST.COM'} at 127.0.0.1:37664
I20260924 16:06:26.935209 11644 tablet_replica.cc:330] T 1eb3bc371dcf4bc985ce6ea040c74038 P acd3c54a49cd4a6ebe2cbd2a34ea80af: stopping tablet replica
I20260924 16:06:26.935509 11644 raft_consensus.cc:2229] T 1eb3bc371dcf4bc985ce6ea040c74038 P acd3c54a49cd4a6ebe2cbd2a34ea80af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:06:26.935981 11644 raft_consensus.cc:2258] T 1eb3bc371dcf4bc985ce6ea040c74038 P acd3c54a49cd4a6ebe2cbd2a34ea80af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:26.936877 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 10998
I20260924 16:06:26.939589 11643 tablet_replica.cc:330] T 1eb3bc371dcf4bc985ce6ea040c74038 P 91edf6392c294ba1abeb6a0ffca2a21f: stopping tablet replica
I20260924 16:06:26.939919 11643 raft_consensus.cc:2229] T 1eb3bc371dcf4bc985ce6ea040c74038 P 91edf6392c294ba1abeb6a0ffca2a21f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:06:26.940346 11643 raft_consensus.cc:2258] T 1eb3bc371dcf4bc985ce6ea040c74038 P 91edf6392c294ba1abeb6a0ffca2a21f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:26.941748 11643 ts_tablet_manager.cc:1905] T 1eb3bc371dcf4bc985ce6ea040c74038 P 91edf6392c294ba1abeb6a0ffca2a21f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:26.943578 11644 ts_tablet_manager.cc:1905] T 1eb3bc371dcf4bc985ce6ea040c74038 P acd3c54a49cd4a6ebe2cbd2a34ea80af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:26.945870 11645 tablet_replica.cc:330] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9: stopping tablet replica
I20260924 16:06:26.946496 11645 raft_consensus.cc:2229] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:06:26.947551 11645 raft_consensus.cc:2258] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:26.948979 11643 ts_tablet_manager.cc:1918] T 1eb3bc371dcf4bc985ce6ea040c74038 P 91edf6392c294ba1abeb6a0ffca2a21f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:06:26.949043 11645 ts_tablet_manager.cc:1905] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:26.954366 11644 ts_tablet_manager.cc:1918] T 1eb3bc371dcf4bc985ce6ea040c74038 P acd3c54a49cd4a6ebe2cbd2a34ea80af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:06:26.954607 11644 log.cc:1196] T 1eb3bc371dcf4bc985ce6ea040c74038 P acd3c54a49cd4a6ebe2cbd2a34ea80af: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/1eb3bc371dcf4bc985ce6ea040c74038
I20260924 16:06:26.955128 11644 ts_tablet_manager.cc:1939] T 1eb3bc371dcf4bc985ce6ea040c74038 P acd3c54a49cd4a6ebe2cbd2a34ea80af: Deleting consensus metadata
I20260924 16:06:26.955629 11643 log.cc:1196] T 1eb3bc371dcf4bc985ce6ea040c74038 P 91edf6392c294ba1abeb6a0ffca2a21f: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/1eb3bc371dcf4bc985ce6ea040c74038
I20260924 16:06:26.956254 11643 ts_tablet_manager.cc:1939] T 1eb3bc371dcf4bc985ce6ea040c74038 P 91edf6392c294ba1abeb6a0ffca2a21f: Deleting consensus metadata
I20260924 16:06:26.956629 11018 catalog_manager.cc:4841] TS acd3c54a49cd4a6ebe2cbd2a34ea80af (127.3.214.194:36209): tablet 1eb3bc371dcf4bc985ce6ea040c74038 (table test-table [id=f608ddc5208a4ee79a4697e318830ab1]) successfully deleted
I20260924 16:06:26.957764 11645 ts_tablet_manager.cc:1918] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:06:26.957963 11017 catalog_manager.cc:4841] TS 91edf6392c294ba1abeb6a0ffca2a21f (127.3.214.195:37151): tablet 1eb3bc371dcf4bc985ce6ea040c74038 (table test-table [id=f608ddc5208a4ee79a4697e318830ab1]) successfully deleted
I20260924 16:06:26.958051 11645 log.cc:1196] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790265920616583-3931-0/minicluster-data/ts-0/wal/wals/1eb3bc371dcf4bc985ce6ea040c74038
I20260924 16:06:26.958722 11645 ts_tablet_manager.cc:1939] T 1eb3bc371dcf4bc985ce6ea040c74038 P caa4aecd532a4f39aa952bd88e021aa9: Deleting consensus metadata
I20260924 16:06:26.960232 11018 catalog_manager.cc:4841] TS caa4aecd532a4f39aa952bd88e021aa9 (127.3.214.193:37329): tablet 1eb3bc371dcf4bc985ce6ea040c74038 (table test-table [id=f608ddc5208a4ee79a4697e318830ab1]) successfully deleted
I20260924 16:06:26.970824 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:26.937940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37692 (local address 127.3.214.193:37329)
0924 16:06:26.938170 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:26.938177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:26.949688 (+ 11511us) server_negotiation.cc:407] Connection header received
0924 16:06:26.950017 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:26.950024 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:26.950143 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:26.950227 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:26.952432 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.953695 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.957767 (+ 4072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:26.958632 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:26.958767 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:26.962896 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:26.962918 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:26.962944 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:26.963015 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:26.966739 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:26.967940 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:26.967951 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:26.967954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:26.968169 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:26.968842 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:26.968847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:26.968849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:26.969376 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:26.969779 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:26.970186 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 16:06:26.970436 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:06:26.971930 11209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:27.048237 11603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:27.047710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:32774)
0924 16:06:27.047944 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:27.048046 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 16:06:27.050028 11603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:27.049379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:32780)
0924 16:06:27.049663 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:27.049815 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 16:06:27.092934 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 10998
I20260924 16:06:27.115304 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 11226
I20260924 16:06:27.138834 11616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:27.116373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52456 (local address 127.3.214.194:36209)
0924 16:06:27.116590 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:27.116598 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:27.116622 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:06:27.116725 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:27.116729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:27.116815 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:27.116923 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:27.118389 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:27.119197 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:27.120608 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:27.121108 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:27.121237 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:27.131433 (+ 10196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:27.131458 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:27.131482 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:27.131540 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:27.135173 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:27.136261 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:27.136271 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:27.136274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:27.136441 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:27.137009 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:27.137014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:27.137016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:27.137484 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:27.137813 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:27.138255 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 16:06:27.138486 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 16:06:27.139869 11358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:27.280434 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 11226
I20260924 16:06:27.306190 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 11378
I20260924 16:06:27.321987 11618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:27.307117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39842 (local address 127.3.214.195:37151)
0924 16:06:27.307376 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:27.307381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:27.307403 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:06:27.307633 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:27.307641 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:27.307724 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:27.307845 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:27.309441 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:27.310381 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:27.311905 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:27.312398 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:27.312530 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:27.315415 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:27.315436 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:27.315459 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:27.315509 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:27.319083 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:27.319886 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:27.319895 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:27.319897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:27.320072 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:27.320459 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:27.320464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:27.320466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:27.320896 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:27.321169 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:27.321433 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 16:06:27.321637 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 16:06:27.322697 11503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:27.427290 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 11378
I20260924 16:06:27.446362 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 10693
I20260924 16:06:27.447710 10866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:27.465871 11603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:27.465272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:32782)
0924 16:06:27.465529 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:27.465687 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 16:06:27.467063 11603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:27.466733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:43105 (local address 127.0.0.1:32798)
0924 16:06:27.466877 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:27.466948 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 16:06:27.573130 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 10693
I20260924 16:06:27.598752 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 10799
I20260924 16:06:27.600312 11063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:27.732302 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 10799
2026-09-24T16:06:27Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12466 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:27 dist-test-slave-1gc3 krb5kdc[11655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:27 dist-test-slave-1gc3 krb5kdc[11655](info): set up 2 sockets
Sep 24 16:06:27 dist-test-slave-1gc3 krb5kdc[11655](info): commencing operation
krb5kdc: starting...
W20260924 16:06:28.793135 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:06:28 dist-test-slave-1gc3 krb5kdc[11655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:28Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.254.keytab.
Entry for principal oryx/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.254.keytab.
I20260924 16:06:29.062700 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:39763
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.254.keytab
--principal=oryx/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:35869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:39763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:29.480077 11668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:29.480631 11668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:29.480803 11668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:29.490175 11668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:29.490384 11668 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 16:06:29.490461 11668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:29.490573 11668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:29.490656 11668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:29.490762 11668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:29.503391 11668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:35869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:39763
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.254.keytab
--principal=oryx/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:39763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:29.505463 11668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:29.507436 11668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:29.522641 11674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:29.522701 11676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:29.522789 11673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:29.528311 11668 server_base.cc:1034] running on GCE node
I20260924 16:06:29.529564 11668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:29.531826 11668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:29.533209 11668 hybrid_clock.cc:648] HybridClock initialized: now 1790265989533122 us; error 57 us; skew 500 ppm
I20260924 16:06:29.533677 11668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:29 dist-test-slave-1gc3 krb5kdc[11655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265989, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:29.538034 11668 init.cc:377] Logged in from keytab as oryx/127.3.214.254@KRBTEST.COM (short username oryx)
I20260924 16:06:29.540414 11668 webserver.cc:460] Webserver started at http://127.3.214.254:33807/ using document root <none> and password file <none>
I20260924 16:06:29.541512 11668 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:29.541633 11668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:29.541990 11668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:29.545783 11668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "623e994a63f541cbb502fa94e1be836a"
format_stamp: "Formatted at 2026-09-24 16:06:29 on dist-test-slave-1gc3"
server_key: "017e630f86db3004ad1da7f14c1fdb37"
server_key_iv: "72a018cce72a916fa61299401916cc2e"
server_key_version: "encryptionkey@0"
I20260924 16:06:29.546818 11668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "623e994a63f541cbb502fa94e1be836a"
format_stamp: "Formatted at 2026-09-24 16:06:29 on dist-test-slave-1gc3"
server_key: "017e630f86db3004ad1da7f14c1fdb37"
server_key_iv: "72a018cce72a916fa61299401916cc2e"
server_key_version: "encryptionkey@0"
I20260924 16:06:29.553640 11668 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 16:06:29.559243 11684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:29.561509 11668 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.004s
I20260924 16:06:29.561843 11668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "623e994a63f541cbb502fa94e1be836a"
format_stamp: "Formatted at 2026-09-24 16:06:29 on dist-test-slave-1gc3"
server_key: "017e630f86db3004ad1da7f14c1fdb37"
server_key_iv: "72a018cce72a916fa61299401916cc2e"
server_key_version: "encryptionkey@0"
I20260924 16:06:29.562179 11668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:29.588997 11668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:29.594965 11668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:29.595300 11668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:29.623605 11668 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:39763
I20260924 16:06:29.623615 11745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:39763 every 8 connection(s)
I20260924 16:06:29.625476 11668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:29.630200 11746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:29.633188 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 11668
I20260924 16:06:29.633440 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:06:29.634284 3931 external_mini_cluster.cc:1428] Setting key 2b544925acf11a2e87378ddb6635f11d
I20260924 16:06:29.645382 11746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a: Bootstrap starting.
I20260924 16:06:29.649902 11746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:29.651423 11746 log.cc:824] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:06:29 dist-test-slave-1gc3 krb5kdc[11655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM
I20260924 16:06:29.655851 11746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a: No bootstrap required, opened a new log
I20260924 16:06:29.664012 11749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:29.637190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40106 (local address 127.3.214.254:39763)
0924 16:06:29.638298 (+ 1108us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:29.638314 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:29.638373 (+ 59us) server_negotiation.cc:407] Connection header received
0924 16:06:29.640517 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:29.640548 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:29.641109 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:29.641553 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:29.643647 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:29.645378 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:29.647352 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:29.648289 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:29.648476 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:29.653697 (+ 5221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:29.653711 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:29.653738 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:29.653794 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:29.658157 (+ 4363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:29.659173 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:29.659187 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:29.659198 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:29.659438 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:29.659894 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:29.659900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:29.659904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:29.660802 (+ 898us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:29.661193 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:29.661645 (+ 452us) server_negotiation.cc:299] Negotiation successful
0924 16:06:29.662273 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":513,"threads_started":1}
I20260924 16:06:29.667356 11746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "623e994a63f541cbb502fa94e1be836a" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39763 } }
I20260924 16:06:29.667905 11746 raft_consensus.cc:374] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:29.668040 11746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e994a63f541cbb502fa94e1be836a, State: Initialized, Role: FOLLOWER
I20260924 16:06:29.668764 11746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "623e994a63f541cbb502fa94e1be836a" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39763 } }
I20260924 16:06:29.669015 11746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:29.669126 11746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:29.669283 11746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:29.671859 11746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "623e994a63f541cbb502fa94e1be836a" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39763 } }
I20260924 16:06:29.672493 11746 leader_election.cc:302] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 623e994a63f541cbb502fa94e1be836a; no voters:
I20260924 16:06:29.673303 11746 leader_election.cc:288] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:29.673483 11751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:29.674777 11751 raft_consensus.cc:686] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a [term 1 LEADER]: Becoming Leader. State: Replica: 623e994a63f541cbb502fa94e1be836a, State: Running, Role: LEADER
I20260924 16:06:29.675504 11751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "623e994a63f541cbb502fa94e1be836a" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39763 } }
I20260924 16:06:29.675981 11746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:29.678923 11752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "623e994a63f541cbb502fa94e1be836a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "623e994a63f541cbb502fa94e1be836a" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39763 } } }
I20260924 16:06:29.679157 11753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 623e994a63f541cbb502fa94e1be836a. Latest consensus state: current_term: 1 leader_uuid: "623e994a63f541cbb502fa94e1be836a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "623e994a63f541cbb502fa94e1be836a" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39763 } } }
I20260924 16:06:29.679477 11752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:29.679482 11753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:29.680544 11756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:29.687881 11756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:29.698257 11768 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 16:06:29.702761 11756 catalog_manager.cc:1347] Generated new cluster ID: 2a53ed7828584400a7ae19dbd4e62590
I20260924 16:06:29.702943 11756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:29.736246 11756 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:29.738616 11756 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:29.747640 11756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a: Generated new TSK 0
I20260924 16:06:29.749024 11756 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.214.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.214.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.214.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.193.keytab.
Entry for principal oryx/127.3.214.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.193.keytab.
I20260924 16:06:29.827850 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39763
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.193.keytab
--principal=oryx/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:35869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:30.256383 11772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:30.256779 11772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:30.256919 11772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:30.265220 11772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:30.265383 11772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:30.265499 11772 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:06:30.265633 11772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:06:30.277266 11772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:35869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.193.keytab
--principal=oryx/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.214.254:39763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:30.279121 11772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:30.281020 11772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:30.297530 11778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:30.297612 11780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:30.298164 11777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:30.298686 11772 server_base.cc:1034] running on GCE node
I20260924 16:06:30.299542 11772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:30.300663 11772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:30.301966 11772 hybrid_clock.cc:648] HybridClock initialized: now 1790265990301863 us; error 70 us; skew 500 ppm
I20260924 16:06:30.302414 11772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:30 dist-test-slave-1gc3 krb5kdc[11655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265990, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:30.306463 11772 init.cc:377] Logged in from keytab as oryx/127.3.214.193@KRBTEST.COM (short username oryx)
I20260924 16:06:30.308422 11772 webserver.cc:460] Webserver started at http://127.3.214.193:36835/ using document root <none> and password file <none>
I20260924 16:06:30.309293 11772 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:30.309437 11772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:30.309828 11772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:30.313546 11772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "78592a6111ce4606a8f33eed3d2ba83b"
format_stamp: "Formatted at 2026-09-24 16:06:30 on dist-test-slave-1gc3"
server_key: "ff0e78f1a26dd07b18fe448a80eb62f0"
server_key_iv: "1f528b7b38a4c854e860782434131e56"
server_key_version: "encryptionkey@0"
I20260924 16:06:30.314563 11772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "78592a6111ce4606a8f33eed3d2ba83b"
format_stamp: "Formatted at 2026-09-24 16:06:30 on dist-test-slave-1gc3"
server_key: "ff0e78f1a26dd07b18fe448a80eb62f0"
server_key_iv: "1f528b7b38a4c854e860782434131e56"
server_key_version: "encryptionkey@0"
I20260924 16:06:30.321650 11772 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260924 16:06:30.326514 11787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:30.328445 11772 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:06:30.328671 11772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "78592a6111ce4606a8f33eed3d2ba83b"
format_stamp: "Formatted at 2026-09-24 16:06:30 on dist-test-slave-1gc3"
server_key: "ff0e78f1a26dd07b18fe448a80eb62f0"
server_key_iv: "1f528b7b38a4c854e860782434131e56"
server_key_version: "encryptionkey@0"
I20260924 16:06:30.328966 11772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:30.351883 11772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:30.356637 11772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:30.356936 11772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:30.360370 11772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:30.360500 11772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:30.360638 11772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:30.360716 11772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:06:30 dist-test-slave-1gc3 krb5kdc[11655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265990, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM
I20260924 16:06:30.390159 11796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:30.361650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:36197 (local address 127.3.214.254:39763)
0924 16:06:30.362272 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:30.362277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:30.363855 (+ 1578us) server_negotiation.cc:407] Connection header received
0924 16:06:30.366909 (+ 3054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:30.366917 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:30.367053 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:30.367235 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:30.370821 (+ 3586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:30.371945 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:30.374129 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:30.374863 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:30.375010 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:30.380610 (+ 5600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:30.380622 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:30.380628 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:30.380732 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:30.384618 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:30.386184 (+ 1566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:30.386191 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:30.386194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:30.386396 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:30.387169 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:30.387175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:30.387179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:30.387638 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:30.387802 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:30.389348 (+ 1546us) server_negotiation.cc:299] Negotiation successful
0924 16:06:30.389692 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":314,"threads_started":1}
I20260924 16:06:30.392491 11797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:30.362444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39763 (local address 127.3.214.193:36197)
0924 16:06:30.363595 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:30.363679 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:30.366371 (+ 2692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:30.367929 (+ 1558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:30.367950 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:30.369132 (+ 1182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:30.370481 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:30.370528 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:30.372158 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:30.372165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:30.373890 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:30.373911 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:30.375249 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:30.375260 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:30.375519 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:30.376500 (+ 981us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:30.376551 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:30.380179 (+ 3628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:30.384935 (+ 4756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:30.384956 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:30.384984 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:30.385924 (+ 940us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:30.386557 (+ 633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:30.386564 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:30.386568 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:30.387026 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:30.387858 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:30.387873 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:30.388667 (+ 794us) client_negotiation.cc:769] Sending connection context
0924 16:06:30.389177 (+ 510us) client_negotiation.cc:240] Negotiation successful
0924 16:06:30.390036 (+ 859us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":459,"threads_started":1}
I20260924 16:06:30.420861 11772 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:42201
I20260924 16:06:30.420939 11908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:42201 every 8 connection(s)
I20260924 16:06:30.422493 11772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:06:30.425395 11909 heartbeater.cc:347] Connected to a master server at 127.3.214.254:39763
I20260924 16:06:30.425757 11909 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:30.426347 11796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:30.404033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:60625 (local address 127.3.214.254:39763)
0924 16:06:30.404282 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:30.404290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:30.408496 (+ 4206us) server_negotiation.cc:407] Connection header received
0924 16:06:30.408861 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:30.408869 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:30.408982 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:30.409076 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:30.410932 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:30.411968 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:30.414734 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:30.415499 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:30.415704 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:30.418928 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:30.418934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:30.418939 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:30.418987 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:30.422418 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:30.423477 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:30.423483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:30.423487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:30.423679 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:30.424383 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:30.424391 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:30.424395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:30.424903 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:30.425077 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:30.425692 (+ 615us) server_negotiation.cc:299] Negotiation successful
0924 16:06:30.425991 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 16:06:30.426563 11868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:30.403762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39763 (local address 127.3.214.193:60625)
0924 16:06:30.408257 (+ 4495us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:30.408293 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:30.408572 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:30.409369 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:30.409375 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:30.410240 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:30.410713 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:30.410729 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:30.412189 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:30.412196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:30.414460 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:30.414488 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:30.415812 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:30.415825 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:30.416060 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:30.417106 (+ 1046us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:30.417138 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:30.418630 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:30.422607 (+ 3977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:30.422614 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:30.422618 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:30.423263 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:30.423861 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:30.423867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:30.423871 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:30.424187 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:30.425056 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:30.425061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:30.425333 (+ 272us) client_negotiation.cc:769] Sending connection context
0924 16:06:30.425525 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 16:06:30.425988 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4221,"thread_start_us":3945,"threads_started":1}
I20260924 16:06:30.426999 11909 heartbeater.cc:511] Master 127.3.214.254:39763 requested a full tablet report, sending...
I20260924 16:06:30.430725 11701 ts_manager.cc:194] Registered new tserver with Master: 78592a6111ce4606a8f33eed3d2ba83b (127.3.214.193:42201)
I20260924 16:06:30.432260 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 11772
I20260924 16:06:30.432461 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:06:30.432508 11701 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:36197
I20260924 16:06:30.433200 3931 external_mini_cluster.cc:1428] Setting key d52452db8847fa5132d46ea0aac148da
WARNING: no policy specified for oryx/127.3.214.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.214.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.214.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.194.keytab.
Entry for principal oryx/127.3.214.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.194.keytab.
I20260924 16:06:30.491060 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39763
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.194.keytab
--principal=oryx/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:35869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:30.891836 11913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:30.892216 11913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:30.892377 11913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:30.900193 11913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:30.900360 11913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:30.900490 11913 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:06:30.900609 11913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:06:30.911453 11913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:35869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.194.keytab
--principal=oryx/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.214.254:39763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:30.912914 11913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:30.914471 11913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:30.930456 11919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:30.930543 11918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:30.930533 11921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:30.931734 11913 server_base.cc:1034] running on GCE node
I20260924 16:06:30.932468 11913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:30.933883 11913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:30.935211 11913 hybrid_clock.cc:648] HybridClock initialized: now 1790265990935086 us; error 84 us; skew 500 ppm
I20260924 16:06:30.935657 11913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:30 dist-test-slave-1gc3 krb5kdc[11655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265990, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:30.939505 11913 init.cc:377] Logged in from keytab as oryx/127.3.214.194@KRBTEST.COM (short username oryx)
I20260924 16:06:30.941373 11913 webserver.cc:460] Webserver started at http://127.3.214.194:33621/ using document root <none> and password file <none>
I20260924 16:06:30.942209 11913 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:30.942317 11913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:30.942644 11913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:30.945925 11913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "575a2d7c7d554b3fbbe872b0a4601ca1"
format_stamp: "Formatted at 2026-09-24 16:06:30 on dist-test-slave-1gc3"
server_key: "3afef94b06e21327a90a99c3b1f7984d"
server_key_iv: "120e45ec1bd3bf0d7cfdb959d3733fd2"
server_key_version: "encryptionkey@0"
I20260924 16:06:30.947002 11913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "575a2d7c7d554b3fbbe872b0a4601ca1"
format_stamp: "Formatted at 2026-09-24 16:06:30 on dist-test-slave-1gc3"
server_key: "3afef94b06e21327a90a99c3b1f7984d"
server_key_iv: "120e45ec1bd3bf0d7cfdb959d3733fd2"
server_key_version: "encryptionkey@0"
I20260924 16:06:30.953374 11913 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 16:06:30.958164 11928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:30.960119 11913 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:06:30.960338 11913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "575a2d7c7d554b3fbbe872b0a4601ca1"
format_stamp: "Formatted at 2026-09-24 16:06:30 on dist-test-slave-1gc3"
server_key: "3afef94b06e21327a90a99c3b1f7984d"
server_key_iv: "120e45ec1bd3bf0d7cfdb959d3733fd2"
server_key_version: "encryptionkey@0"
I20260924 16:06:30.960609 11913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:30.973946 11913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:30.979293 11913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:30.979595 11913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:30.983067 11913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:30.983170 11913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:30.983336 11913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:30.983422 11913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:06:31 dist-test-slave-1gc3 krb5kdc[11655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265990, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM
I20260924 16:06:31.013870 11937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:30.984218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:39217 (local address 127.3.214.254:39763)
0924 16:06:30.984746 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:30.984751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:30.986580 (+ 1829us) server_negotiation.cc:407] Connection header received
0924 16:06:30.989500 (+ 2920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:30.989507 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:30.989626 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:30.989743 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:30.992656 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:30.993796 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:30.996705 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:30.997356 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:30.997505 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:31.004320 (+ 6815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:31.004329 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:31.004335 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:31.004399 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:31.008475 (+ 4076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:31.010132 (+ 1657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:31.010139 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:31.010142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:31.010349 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:31.011096 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:31.011103 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:31.011105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:31.011533 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:31.011733 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:31.013034 (+ 1301us) server_negotiation.cc:299] Negotiation successful
0924 16:06:31.013385 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":289,"threads_started":1}
I20260924 16:06:31.015489 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:30.985296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39763 (local address 127.3.214.194:39217)
0924 16:06:30.986367 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:30.986421 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:30.988908 (+ 2487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:30.990179 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:30.990192 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:30.991189 (+ 997us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:30.992345 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:30.992389 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:30.993974 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:30.993979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:30.996507 (+ 2528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:30.996535 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:30.997701 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:30.997711 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:30.998043 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:30.999135 (+ 1092us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:30.999182 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:31.003788 (+ 4606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:31.008791 (+ 5003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:31.008812 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:31.008849 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:31.009869 (+ 1020us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:31.010520 (+ 651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:31.010526 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:31.010533 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:31.010910 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:31.011734 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:31.011745 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:31.012356 (+ 611us) client_negotiation.cc:769] Sending connection context
0924 16:06:31.012757 (+ 401us) client_negotiation.cc:240] Negotiation successful
0924 16:06:31.013529 (+ 772us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":439,"threads_started":1}
I20260924 16:06:31.040858 11913 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:43513
I20260924 16:06:31.040890 12049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:43513 every 8 connection(s)
I20260924 16:06:31.042397 11913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:06:31.042786 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 11913
I20260924 16:06:31.042903 11937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:31.024364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:54457 (local address 127.3.214.254:39763)
0924 16:06:31.024663 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:31.024672 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:31.025698 (+ 1026us) server_negotiation.cc:407] Connection header received
0924 16:06:31.025983 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:31.025989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:31.026078 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:31.026226 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:31.028337 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:31.029709 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:31.031814 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:31.032368 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:31.032501 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:31.035212 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:31.035220 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:31.035225 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:31.035283 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:31.039200 (+ 3917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:31.040265 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:31.040272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:31.040275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:31.040524 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:31.041008 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:31.041013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:31.041017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:31.041465 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:31.041645 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:31.042191 (+ 546us) server_negotiation.cc:299] Negotiation successful
0924 16:06:31.042492 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260924 16:06:31.042971 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:06:31.043013 12017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:31.024556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39763 (local address 127.3.214.194:54457)
0924 16:06:31.025493 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:31.025524 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:31.025793 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:31.026513 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:31.026519 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:31.027498 (+ 979us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:31.028073 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:31.028090 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:31.029930 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:31.029935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:31.031612 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:31.031629 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:31.032589 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:31.032595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:31.032799 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:31.033652 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:31.033672 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:31.034924 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:31.039397 (+ 4473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:31.039404 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:31.039407 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:31.040102 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:31.040682 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:31.040688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:31.040693 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:31.040881 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:31.041593 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:31.041598 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:31.041831 (+ 233us) client_negotiation.cc:769] Sending connection context
0924 16:06:31.042002 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 16:06:31.042534 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":491,"threads_started":1}
I20260924 16:06:31.043527 3931 external_mini_cluster.cc:1428] Setting key 10d4d3612cc8390d8320b3e99bddb267
I20260924 16:06:31.045724 12050 heartbeater.cc:347] Connected to a master server at 127.3.214.254:39763
I20260924 16:06:31.045987 12050 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:31.047215 12050 heartbeater.cc:511] Master 127.3.214.254:39763 requested a full tablet report, sending...
I20260924 16:06:31.049512 11701 ts_manager.cc:194] Registered new tserver with Master: 575a2d7c7d554b3fbbe872b0a4601ca1 (127.3.214.194:43513)
I20260924 16:06:31.050525 11701 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:39217
WARNING: no policy specified for oryx/127.3.214.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.214.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.214.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.195.keytab.
Entry for principal oryx/127.3.214.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.195.keytab.
I20260924 16:06:31.102737 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39763
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.195.keytab
--principal=oryx/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:35869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:06:31.436602 11909 heartbeater.cc:503] Master 127.3.214.254:39763 was elected leader, sending a full tablet report...
W20260924 16:06:31.505535 12054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:31.505942 12054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:31.506079 12054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:31.514361 12054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:31.514533 12054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:31.514681 12054 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:06:31.514820 12054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:06:31.526397 12054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:35869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.195.keytab
--principal=oryx/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.214.254:39763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:31.528156 12054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:31.529892 12054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:31.546212 12059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:31.546576 12060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:31.547772 12062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:31.547765 12054 server_base.cc:1034] running on GCE node
I20260924 16:06:31.548704 12054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:31.549902 12054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:31.551218 12054 hybrid_clock.cc:648] HybridClock initialized: now 1790265991551121 us; error 70 us; skew 500 ppm
I20260924 16:06:31.551635 12054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:31 dist-test-slave-1gc3 krb5kdc[11655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265991, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:31.555609 12054 init.cc:377] Logged in from keytab as oryx/127.3.214.195@KRBTEST.COM (short username oryx)
I20260924 16:06:31.557574 12054 webserver.cc:460] Webserver started at http://127.3.214.195:40151/ using document root <none> and password file <none>
I20260924 16:06:31.558568 12054 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:31.558691 12054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:31.559062 12054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:31.562848 12054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "a36b510d73684a5ea8a77d4745925790"
format_stamp: "Formatted at 2026-09-24 16:06:31 on dist-test-slave-1gc3"
server_key: "c2e6ea8e43371a3f4549a3cc8cf5c099"
server_key_iv: "d19db9b1d1f98eb5e37b8c5a9686eba6"
server_key_version: "encryptionkey@0"
I20260924 16:06:31.563823 12054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "a36b510d73684a5ea8a77d4745925790"
format_stamp: "Formatted at 2026-09-24 16:06:31 on dist-test-slave-1gc3"
server_key: "c2e6ea8e43371a3f4549a3cc8cf5c099"
server_key_iv: "d19db9b1d1f98eb5e37b8c5a9686eba6"
server_key_version: "encryptionkey@0"
I20260924 16:06:31.570794 12054 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260924 16:06:31.575608 12069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:31.577868 12054 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:06:31.578094 12054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "a36b510d73684a5ea8a77d4745925790"
format_stamp: "Formatted at 2026-09-24 16:06:31 on dist-test-slave-1gc3"
server_key: "c2e6ea8e43371a3f4549a3cc8cf5c099"
server_key_iv: "d19db9b1d1f98eb5e37b8c5a9686eba6"
server_key_version: "encryptionkey@0"
I20260924 16:06:31.578411 12054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:31.595113 12054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:31.600392 12054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:31.600713 12054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:31.604210 12054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:31.604311 12054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:31.604471 12054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:31.604542 12054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:06:31 dist-test-slave-1gc3 krb5kdc[11655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265991, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM
I20260924 16:06:31.633615 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:31.604758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:55063 (local address 127.3.214.254:39763)
0924 16:06:31.605635 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:31.605641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:31.606904 (+ 1263us) server_negotiation.cc:407] Connection header received
0924 16:06:31.609338 (+ 2434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:31.609346 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:31.609491 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:31.609693 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:31.612521 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:31.613611 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:31.616511 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:31.617017 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:31.617130 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:31.624043 (+ 6913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:31.624052 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:31.624057 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:31.624113 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:31.628331 (+ 4218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:31.629801 (+ 1470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:31.629808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:31.629812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:31.630042 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:31.630923 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:31.630928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:31.630931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:31.631341 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:31.631506 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:31.632952 (+ 1446us) server_negotiation.cc:299] Negotiation successful
0924 16:06:31.633203 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":537,"threads_started":1}
I20260924 16:06:31.636176 12079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:31.605549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39763 (local address 127.3.214.195:55063)
0924 16:06:31.606685 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:31.606735 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:31.608939 (+ 2204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:31.610075 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:31.610094 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:31.611153 (+ 1059us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:31.612224 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:31.612261 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:31.613787 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:31.613792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:31.616240 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:31.616258 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:31.617353 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:31.617366 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:31.617854 (+ 488us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:31.619065 (+ 1211us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:31.619115 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:31.623568 (+ 4453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:31.628598 (+ 5030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:31.628615 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:31.628646 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:31.629574 (+ 928us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:31.630304 (+ 730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:31.630312 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:31.630316 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:31.630726 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:31.631526 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:31.631539 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:31.632344 (+ 805us) client_negotiation.cc:769] Sending connection context
0924 16:06:31.632786 (+ 442us) client_negotiation.cc:240] Negotiation successful
0924 16:06:31.633664 (+ 878us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":483,"threads_started":1}
I20260924 16:06:31.665212 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:31.643804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39763 (local address 127.3.214.195:42279)
0924 16:06:31.644682 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:31.644713 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:31.644965 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:31.645502 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:31.645507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:31.646380 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:31.647375 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:31.647405 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:31.649057 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:31.649063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:31.651257 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:31.651288 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:31.652511 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:31.652522 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:31.652759 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:31.653697 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:31.653729 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:31.655400 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:31.660356 (+ 4956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:31.660363 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:31.660368 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:31.661230 (+ 862us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:31.661956 (+ 726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:31.661965 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:31.661970 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:31.662300 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:31.663554 (+ 1254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:31.663559 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:31.663845 (+ 286us) client_negotiation.cc:769] Sending connection context
0924 16:06:31.664053 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 16:06:31.664609 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":433,"threads_started":1}
I20260924 16:06:31.666160 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:31.643793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:42279 (local address 127.3.214.254:39763)
0924 16:06:31.644078 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:31.644087 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:31.644909 (+ 822us) server_negotiation.cc:407] Connection header received
0924 16:06:31.645183 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:31.645188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:31.645273 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:31.645427 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:31.647817 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:31.648790 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:31.651688 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:31.652220 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:31.652350 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:31.655780 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:31.655787 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:31.655791 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:31.655846 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:31.660056 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:31.661494 (+ 1438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:31.661499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:31.661503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:31.661708 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:31.662585 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:31.662592 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:31.662595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:31.663279 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:31.663483 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:31.665645 (+ 2162us) server_negotiation.cc:299] Negotiation successful
0924 16:06:31.665886 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260924 16:06:31.669893 12054 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:37245
I20260924 16:06:31.669898 12191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:37245 every 8 connection(s)
I20260924 16:06:31.671832 12054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:06:31.672271 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 12054
I20260924 16:06:31.672454 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:06:31.673022 3931 external_mini_cluster.cc:1428] Setting key e8ccc0a4691d30156f6389e6a6dfeab3
I20260924 16:06:31.674700 12192 heartbeater.cc:347] Connected to a master server at 127.3.214.254:39763
I20260924 16:06:31.675010 12192 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:31.676026 12192 heartbeater.cc:511] Master 127.3.214.254:39763 requested a full tablet report, sending...
I20260924 16:06:31.677742 11700 ts_manager.cc:194] Registered new tserver with Master: a36b510d73684a5ea8a77d4745925790 (127.3.214.195:37245)
I20260924 16:06:31.678828 11700 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:55063
I20260924 16:06:31.690148 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 16:06:31 dist-test-slave-1gc3 krb5kdc[11655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265991, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 16:06:31 dist-test-slave-1gc3 krb5kdc[11655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:31 dist-test-slave-1gc3 krb5kdc[11655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:32.054036 12050 heartbeater.cc:503] Master 127.3.214.254:39763 was elected leader, sending a full tablet report...
I20260924 16:06:32.681734 12192 heartbeater.cc:503] Master 127.3.214.254:39763 was elected leader, sending a full tablet report...
W20260924 16:06:32.800077 12200 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.052s user 0.000s sys 0.017s
W20260924 16:06:32.801646 12078 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:31.741530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40118 (local address 127.3.214.254:39763)
0924 16:06:31.741747 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:31.741751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:31.741944 (+ 193us) server_negotiation.cc:407] Connection header received
0924 16:06:31.742179 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:31.742183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:31.742287 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:31.742467 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:31.743990 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:31.745055 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:31.746583 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:31.747071 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:31.747215 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:32.801287 (+1054072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40118: BlockingRecv error: recv got EOF from 127.0.0.1:40118 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 24 16:06:32 dist-test-slave-1gc3 krb5kdc[11655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265991, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM
I20260924 16:06:32.829977 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.806318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.3.214.254:39763)
0924 16:06:32.806587 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:32.806595 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:32.806801 (+ 206us) server_negotiation.cc:407] Connection header received
0924 16:06:32.807045 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:32.807059 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:32.807144 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:32.807251 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:32.808836 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.809663 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.811170 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.811671 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.811841 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:32.823510 (+ 11669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:32.823518 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:32.823522 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:32.823583 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:32.826864 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:32.827824 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:32.827829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:32.827831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:32.827988 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:32.828384 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:32.828388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:32.828390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:32.828725 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:32.829015 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:32.829341 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 16:06:32.829615 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 16:06:32.835528 11700 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:06:32.843927 11700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:06:32.874078 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.861453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:37245 (local address 127.0.0.1:55858)
0924 16:06:32.863740 (+ 2287us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:32.863795 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:32.864045 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:32.865024 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:32.865031 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:32.865060 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:32.865510 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:32.865526 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.867647 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.867653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:32.870432 (+ 2779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:32.870447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.872948 (+ 2501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.872955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:32.873205 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:32.873273 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 16:06:32.873425 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 16:06:32.873644 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1979,"thread_start_us":1821,"threads_started":1}
I20260924 16:06:32.875262 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.861886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55858 (local address 127.3.214.195:37245)
0924 16:06:32.862459 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:32.862469 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:32.864012 (+ 1543us) server_negotiation.cc:407] Connection header received
0924 16:06:32.864374 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:32.864385 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:32.864774 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:32.865027 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:32.865762 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.867441 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.870667 (+ 3226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.872310 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.872468 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:32.872725 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:32.874728 (+ 2003us) server_negotiation.cc:299] Negotiation successful
0924 16:06:32.874952 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":344,"threads_started":1}
I20260924 16:06:32.878046 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.862020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60260 (local address 127.3.214.194:43513)
0924 16:06:32.863031 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:32.863041 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:32.864816 (+ 1775us) server_negotiation.cc:407] Connection header received
0924 16:06:32.864988 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:32.864997 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:32.865316 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:32.865570 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:32.866641 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.868121 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.874186 (+ 6065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.875826 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.875967 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:32.876252 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:32.877555 (+ 1303us) server_negotiation.cc:299] Negotiation successful
0924 16:06:32.877755 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":617,"threads_started":1}
I20260924 16:06:32.878865 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.861532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:43513 (local address 127.0.0.1:60260)
0924 16:06:32.864670 (+ 3138us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:32.864690 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:32.864824 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:32.865915 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:32.865920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:32.865948 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:32.866433 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:32.866447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.870229 (+ 3782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.870235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:32.872408 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:32.872427 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.877119 (+ 4692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.877126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:32.877306 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:32.877352 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 16:06:32.878430 (+ 1078us) client_negotiation.cc:240] Negotiation successful
0924 16:06:32.878597 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3000,"thread_start_us":2856,"threads_started":1}
I20260924 16:06:32.881065 12120 tablet_service.cc:1467] Processing CreateTablet for tablet 9c4596277b1243a48c9bea5cafbf83ed (DEFAULT_TABLE table=test-table [id=85c5af5ddf7e488c8d0ff1cd5294f28f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:32.883170 12120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c4596277b1243a48c9bea5cafbf83ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:32.883317 11980 tablet_service.cc:1467] Processing CreateTablet for tablet 9c4596277b1243a48c9bea5cafbf83ed (DEFAULT_TABLE table=test-table [id=85c5af5ddf7e488c8d0ff1cd5294f28f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:32.885147 11980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c4596277b1243a48c9bea5cafbf83ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:32.885300 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.864985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:42201 (local address 127.0.0.1:44034)
0924 16:06:32.874397 (+ 9412us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:32.874419 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:32.874528 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:32.877619 (+ 3091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:32.877625 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:32.877682 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:32.877915 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:32.877930 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.880143 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.880148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:32.881980 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:32.882007 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.884451 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.884457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:32.884671 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:32.884712 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 16:06:32.884829 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 16:06:32.884994 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9291,"thread_start_us":10522,"threads_started":1}
I20260924 16:06:32.885489 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:32.865708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44034 (local address 127.3.214.193:42201)
0924 16:06:32.876447 (+ 10739us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:32.876460 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:32.876509 (+ 49us) server_negotiation.cc:407] Connection header received
0924 16:06:32.876640 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:32.876648 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:32.876989 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:32.877320 (+ 331us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:32.878177 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.879944 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.882260 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:32.883992 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:32.884177 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:32.884442 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:32.884918 (+ 476us) server_negotiation.cc:299] Negotiation successful
0924 16:06:32.885127 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10521,"thread_start_us":10345,"threads_started":1}
I20260924 16:06:32.890563 11839 tablet_service.cc:1467] Processing CreateTablet for tablet 9c4596277b1243a48c9bea5cafbf83ed (DEFAULT_TABLE table=test-table [id=85c5af5ddf7e488c8d0ff1cd5294f28f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:32.892724 11839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c4596277b1243a48c9bea5cafbf83ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:32.899483 12217 tablet_bootstrap.cc:492] T 9c4596277b1243a48c9bea5cafbf83ed P 575a2d7c7d554b3fbbe872b0a4601ca1: Bootstrap starting.
I20260924 16:06:32.903034 12218 tablet_bootstrap.cc:492] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790: Bootstrap starting.
I20260924 16:06:32.903429 12217 tablet_bootstrap.cc:654] T 9c4596277b1243a48c9bea5cafbf83ed P 575a2d7c7d554b3fbbe872b0a4601ca1: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:32.905035 12217 log.cc:824] T 9c4596277b1243a48c9bea5cafbf83ed P 575a2d7c7d554b3fbbe872b0a4601ca1: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:32.906888 12218 tablet_bootstrap.cc:654] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:32.908185 12218 log.cc:824] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:32.909652 12217 tablet_bootstrap.cc:492] T 9c4596277b1243a48c9bea5cafbf83ed P 575a2d7c7d554b3fbbe872b0a4601ca1: No bootstrap required, opened a new log
I20260924 16:06:32.910058 12217 ts_tablet_manager.cc:1397] T 9c4596277b1243a48c9bea5cafbf83ed P 575a2d7c7d554b3fbbe872b0a4601ca1: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260924 16:06:32.911367 12221 tablet_bootstrap.cc:492] T 9c4596277b1243a48c9bea5cafbf83ed P 78592a6111ce4606a8f33eed3d2ba83b: Bootstrap starting.
I20260924 16:06:32.911540 12218 tablet_bootstrap.cc:492] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790: No bootstrap required, opened a new log
I20260924 16:06:32.912225 12218 ts_tablet_manager.cc:1397] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 16:06:32.915640 12221 tablet_bootstrap.cc:654] T 9c4596277b1243a48c9bea5cafbf83ed P 78592a6111ce4606a8f33eed3d2ba83b: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:32.917111 12221 log.cc:824] T 9c4596277b1243a48c9bea5cafbf83ed P 78592a6111ce4606a8f33eed3d2ba83b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:32.919775 12217 raft_consensus.cc:348] T 9c4596277b1243a48c9bea5cafbf83ed P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } }
I20260924 16:06:32.920462 12217 raft_consensus.cc:374] T 9c4596277b1243a48c9bea5cafbf83ed P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:32.920590 12217 raft_consensus.cc:729] T 9c4596277b1243a48c9bea5cafbf83ed P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 575a2d7c7d554b3fbbe872b0a4601ca1, State: Initialized, Role: FOLLOWER
I20260924 16:06:32.921408 12217 consensus_queue.cc:260] T 9c4596277b1243a48c9bea5cafbf83ed P 575a2d7c7d554b3fbbe872b0a4601ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } }
I20260924 16:06:32.923422 12221 tablet_bootstrap.cc:492] T 9c4596277b1243a48c9bea5cafbf83ed P 78592a6111ce4606a8f33eed3d2ba83b: No bootstrap required, opened a new log
I20260924 16:06:32.923792 12221 ts_tablet_manager.cc:1397] T 9c4596277b1243a48c9bea5cafbf83ed P 78592a6111ce4606a8f33eed3d2ba83b: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260924 16:06:32.923483 12218 raft_consensus.cc:348] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } }
I20260924 16:06:32.924737 12218 raft_consensus.cc:374] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:32.924888 12218 raft_consensus.cc:729] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a36b510d73684a5ea8a77d4745925790, State: Initialized, Role: FOLLOWER
I20260924 16:06:32.925268 12217 ts_tablet_manager.cc:1428] T 9c4596277b1243a48c9bea5cafbf83ed P 575a2d7c7d554b3fbbe872b0a4601ca1: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260924 16:06:32.925642 12218 consensus_queue.cc:260] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } }
I20260924 16:06:32.928982 12218 ts_tablet_manager.cc:1428] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790: Time spent starting tablet: real 0.017s user 0.003s sys 0.010s
I20260924 16:06:32.934324 12221 raft_consensus.cc:348] T 9c4596277b1243a48c9bea5cafbf83ed P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } }
I20260924 16:06:32.934751 12221 raft_consensus.cc:374] T 9c4596277b1243a48c9bea5cafbf83ed P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:32.934837 12221 raft_consensus.cc:729] T 9c4596277b1243a48c9bea5cafbf83ed P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78592a6111ce4606a8f33eed3d2ba83b, State: Initialized, Role: FOLLOWER
I20260924 16:06:32.935461 12221 consensus_queue.cc:260] T 9c4596277b1243a48c9bea5cafbf83ed P 78592a6111ce4606a8f33eed3d2ba83b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } }
I20260924 16:06:32.937423 12221 ts_tablet_manager.cc:1428] T 9c4596277b1243a48c9bea5cafbf83ed P 78592a6111ce4606a8f33eed3d2ba83b: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260924 16:06:33.047104 12051 tablet.cc:2369] T 9c4596277b1243a48c9bea5cafbf83ed P 575a2d7c7d554b3fbbe872b0a4601ca1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:06:33.139704 12224 raft_consensus.cc:482] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:06:33.140123 12224 raft_consensus.cc:504] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } }
I20260924 16:06:33.142329 12224 leader_election.cc:288] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 575a2d7c7d554b3fbbe872b0a4601ca1 (127.3.214.194:43513), 78592a6111ce4606a8f33eed3d2ba83b (127.3.214.193:42201)
I20260924 16:06:33.151973 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.142955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:57393 (local address 127.3.214.194:43513)
0924 16:06:33.143205 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.143213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.143783 (+ 570us) server_negotiation.cc:407] Connection header received
0924 16:06:33.144136 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.144145 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.144316 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.144439 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:33.145203 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.146239 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.149218 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.150619 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.150795 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.150885 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:33.151528 (+ 643us) server_negotiation.cc:299] Negotiation successful
0924 16:06:33.151683 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:06:33.151973 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.144018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:53807 (local address 127.3.214.193:42201)
0924 16:06:33.144278 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.144284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.144660 (+ 376us) server_negotiation.cc:407] Connection header received
0924 16:06:33.144912 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.144916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.145052 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.145154 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:33.145685 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.146669 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.149222 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.150558 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.150786 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.150890 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:33.151528 (+ 638us) server_negotiation.cc:299] Negotiation successful
0924 16:06:33.151683 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 16:06:33.152275 12226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.142827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:43513 (local address 127.3.214.195:57393)
0924 16:06:33.143646 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:33.143666 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:33.143862 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:33.144500 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:33.144506 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:33.144539 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:33.145035 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.145046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.146419 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.146423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.148910 (+ 2487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.148934 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.150954 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.150960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.151231 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.151336 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 16:06:33.151462 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 16:06:33.151690 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":99,"spinlock_wait_cycles":24960,"thread_start_us":447,"threads_started":1}
I20260924 16:06:33.152316 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.143816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:42201 (local address 127.3.214.195:53807)
0924 16:06:33.144513 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:33.144530 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:33.144766 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:33.145195 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:33.145198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:33.145221 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:33.145528 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.145537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.146829 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.146834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.149007 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.149017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.150945 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.150953 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.151255 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.151336 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 16:06:33.151463 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 16:06:33.151691 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":79,"spinlock_wait_cycles":47488,"thread_start_us":325,"threads_started":1}
I20260924 16:06:33.153947 12000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c4596277b1243a48c9bea5cafbf83ed" candidate_uuid: "a36b510d73684a5ea8a77d4745925790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" is_pre_election: true
I20260924 16:06:33.153978 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c4596277b1243a48c9bea5cafbf83ed" candidate_uuid: "a36b510d73684a5ea8a77d4745925790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78592a6111ce4606a8f33eed3d2ba83b" is_pre_election: true
I20260924 16:06:33.154582 12000 raft_consensus.cc:2454] T 9c4596277b1243a48c9bea5cafbf83ed P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a36b510d73684a5ea8a77d4745925790 in term 0.
I20260924 16:06:33.154634 11863 raft_consensus.cc:2454] T 9c4596277b1243a48c9bea5cafbf83ed P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a36b510d73684a5ea8a77d4745925790 in term 0.
I20260924 16:06:33.155313 12070 leader_election.cc:302] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 575a2d7c7d554b3fbbe872b0a4601ca1, a36b510d73684a5ea8a77d4745925790; no voters:
I20260924 16:06:33.155826 12224 raft_consensus.cc:2781] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:06:33.156018 12224 raft_consensus.cc:482] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:06:33.156117 12224 raft_consensus.cc:3037] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:33.158885 12224 raft_consensus.cc:504] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } }
I20260924 16:06:33.159670 12224 leader_election.cc:288] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790 [CANDIDATE]: Term 1 election: Requested vote from peers 575a2d7c7d554b3fbbe872b0a4601ca1 (127.3.214.194:43513), 78592a6111ce4606a8f33eed3d2ba83b (127.3.214.193:42201)
I20260924 16:06:33.160422 12000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c4596277b1243a48c9bea5cafbf83ed" candidate_uuid: "a36b510d73684a5ea8a77d4745925790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1"
I20260924 16:06:33.160673 12000 raft_consensus.cc:3037] T 9c4596277b1243a48c9bea5cafbf83ed P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:33.160573 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c4596277b1243a48c9bea5cafbf83ed" candidate_uuid: "a36b510d73684a5ea8a77d4745925790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78592a6111ce4606a8f33eed3d2ba83b"
I20260924 16:06:33.160894 11863 raft_consensus.cc:3037] T 9c4596277b1243a48c9bea5cafbf83ed P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:33.163205 12000 raft_consensus.cc:2454] T 9c4596277b1243a48c9bea5cafbf83ed P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36b510d73684a5ea8a77d4745925790 in term 1.
I20260924 16:06:33.163640 11863 raft_consensus.cc:2454] T 9c4596277b1243a48c9bea5cafbf83ed P 78592a6111ce4606a8f33eed3d2ba83b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36b510d73684a5ea8a77d4745925790 in term 1.
I20260924 16:06:33.163786 12070 leader_election.cc:302] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 575a2d7c7d554b3fbbe872b0a4601ca1, a36b510d73684a5ea8a77d4745925790; no voters:
I20260924 16:06:33.164255 12224 raft_consensus.cc:2781] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:33.164816 12224 raft_consensus.cc:686] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790 [term 1 LEADER]: Becoming Leader. State: Replica: a36b510d73684a5ea8a77d4745925790, State: Running, Role: LEADER
I20260924 16:06:33.165556 12224 consensus_queue.cc:237] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } }
I20260924 16:06:33.171741 11700 catalog_manager.cc:5495] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790 reported cstate change: term changed from 0 to 1, leader changed from <none> to a36b510d73684a5ea8a77d4745925790 (127.3.214.195). New cstate: current_term: 1 leader_uuid: "a36b510d73684a5ea8a77d4745925790" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } health_report { overall_health: UNKNOWN } } }
W20260924 16:06:33.174932 12193 tablet.cc:2369] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:06:33.176764 11910 tablet.cc:2369] T 9c4596277b1243a48c9bea5cafbf83ed P 78592a6111ce4606a8f33eed3d2ba83b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:06:33.254302 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.248589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55860 (local address 127.3.214.195:37245)
0924 16:06:33.248886 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.248892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.248917 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:06:33.249038 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.249042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.249156 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.249315 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:33.249723 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.250549 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.251996 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.252419 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.252539 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.252990 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:33.253636 (+ 646us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:33.253750 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:33.253886 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 16:06:33.253969 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260924 16:06:33.265267 12000 raft_consensus.cc:1261] T 9c4596277b1243a48c9bea5cafbf83ed P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 1 FOLLOWER]: Refusing update from remote peer a36b510d73684a5ea8a77d4745925790: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:33.265349 11863 raft_consensus.cc:1261] T 9c4596277b1243a48c9bea5cafbf83ed P 78592a6111ce4606a8f33eed3d2ba83b [term 1 FOLLOWER]: Refusing update from remote peer a36b510d73684a5ea8a77d4745925790: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:33.266434 12224 consensus_queue.cc:1035] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790 [LEADER]: Connected to new peer: Peer: permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:33.266922 12228 consensus_queue.cc:1035] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:33.300809 11700 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:33.301206 11700 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:33.305750 11700 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a: Sending DeleteTablet for 3 replicas of tablet 9c4596277b1243a48c9bea5cafbf83ed
I20260924 16:06:33.307345 12120 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c4596277b1243a48c9bea5cafbf83ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:33 UTC) from {username='oryx', principal='oryx/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:55858
I20260924 16:06:33.307579 11839 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c4596277b1243a48c9bea5cafbf83ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:33 UTC) from {username='oryx', principal='oryx/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:44034
I20260924 16:06:33.308238 11980 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c4596277b1243a48c9bea5cafbf83ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:33 UTC) from {username='oryx', principal='oryx/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:60260
I20260924 16:06:33.309366 11700 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:06:33.310390 11700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:06:33.313779 12241 tablet_replica.cc:330] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790: stopping tablet replica
I20260924 16:06:33.313884 12243 tablet_replica.cc:330] T 9c4596277b1243a48c9bea5cafbf83ed P 575a2d7c7d554b3fbbe872b0a4601ca1: stopping tablet replica
I20260924 16:06:33.314340 12241 raft_consensus.cc:2229] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:06:33.314579 12243 raft_consensus.cc:2229] T 9c4596277b1243a48c9bea5cafbf83ed P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:06:33.315068 12241 raft_consensus.cc:2258] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:33.315248 12243 raft_consensus.cc:2258] T 9c4596277b1243a48c9bea5cafbf83ed P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:33.320870 12241 ts_tablet_manager.cc:1905] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:33.320164 12243 ts_tablet_manager.cc:1905] T 9c4596277b1243a48c9bea5cafbf83ed P 575a2d7c7d554b3fbbe872b0a4601ca1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:33.321979 11838 tablet_service.cc:1467] Processing CreateTablet for tablet eaa254cf54d44e2e87bfee33bd404567 (DEFAULT_TABLE table=test-table [id=3593fd570f144f0b95e6888f79dd2487]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:33.322856 11838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa254cf54d44e2e87bfee33bd404567. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:33.327592 12242 tablet_replica.cc:330] T 9c4596277b1243a48c9bea5cafbf83ed P 78592a6111ce4606a8f33eed3d2ba83b: stopping tablet replica
I20260924 16:06:33.328272 12242 raft_consensus.cc:2229] T 9c4596277b1243a48c9bea5cafbf83ed P 78592a6111ce4606a8f33eed3d2ba83b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:06:33.328958 12242 raft_consensus.cc:2258] T 9c4596277b1243a48c9bea5cafbf83ed P 78592a6111ce4606a8f33eed3d2ba83b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:33.328951 11980 tablet_service.cc:1467] Processing CreateTablet for tablet eaa254cf54d44e2e87bfee33bd404567 (DEFAULT_TABLE table=test-table [id=3593fd570f144f0b95e6888f79dd2487]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:33.329581 11980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa254cf54d44e2e87bfee33bd404567. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:33.335206 12221 tablet_bootstrap.cc:492] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b: Bootstrap starting.
I20260924 16:06:33.337067 12243 ts_tablet_manager.cc:1918] T 9c4596277b1243a48c9bea5cafbf83ed P 575a2d7c7d554b3fbbe872b0a4601ca1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:06:33.337349 12221 tablet_bootstrap.cc:654] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:33.337424 12243 log.cc:1196] T 9c4596277b1243a48c9bea5cafbf83ed P 575a2d7c7d554b3fbbe872b0a4601ca1: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/9c4596277b1243a48c9bea5cafbf83ed
I20260924 16:06:33.337074 12217 tablet_bootstrap.cc:492] T eaa254cf54d44e2e87bfee33bd404567 P 575a2d7c7d554b3fbbe872b0a4601ca1: Bootstrap starting.
I20260924 16:06:33.339025 12242 ts_tablet_manager.cc:1905] T 9c4596277b1243a48c9bea5cafbf83ed P 78592a6111ce4606a8f33eed3d2ba83b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:33.339928 12221 tablet_bootstrap.cc:492] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b: No bootstrap required, opened a new log
I20260924 16:06:33.340116 12221 ts_tablet_manager.cc:1397] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 16:06:33.340927 12221 raft_consensus.cc:348] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:33.341147 12221 raft_consensus.cc:374] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:33.341250 12221 raft_consensus.cc:729] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78592a6111ce4606a8f33eed3d2ba83b, State: Initialized, Role: FOLLOWER
I20260924 16:06:33.341560 12221 consensus_queue.cc:260] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:33.342828 12221 ts_tablet_manager.cc:1428] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 16:06:33.343969 12243 ts_tablet_manager.cc:1939] T 9c4596277b1243a48c9bea5cafbf83ed P 575a2d7c7d554b3fbbe872b0a4601ca1: Deleting consensus metadata
I20260924 16:06:33.345297 12217 tablet_bootstrap.cc:654] T eaa254cf54d44e2e87bfee33bd404567 P 575a2d7c7d554b3fbbe872b0a4601ca1: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:33.345744 12120 tablet_service.cc:1467] Processing CreateTablet for tablet eaa254cf54d44e2e87bfee33bd404567 (DEFAULT_TABLE table=test-table [id=3593fd570f144f0b95e6888f79dd2487]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:33.346920 11685 catalog_manager.cc:4841] TS 575a2d7c7d554b3fbbe872b0a4601ca1 (127.3.214.194:43513): tablet 9c4596277b1243a48c9bea5cafbf83ed (table test-table [id=85c5af5ddf7e488c8d0ff1cd5294f28f]) successfully deleted
I20260924 16:06:33.348419 12241 ts_tablet_manager.cc:1918] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:06:33.348630 12241 log.cc:1196] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/9c4596277b1243a48c9bea5cafbf83ed
I20260924 16:06:33.349040 12241 ts_tablet_manager.cc:1939] T 9c4596277b1243a48c9bea5cafbf83ed P a36b510d73684a5ea8a77d4745925790: Deleting consensus metadata
I20260924 16:06:33.350916 11688 catalog_manager.cc:4841] TS a36b510d73684a5ea8a77d4745925790 (127.3.214.195:37245): tablet 9c4596277b1243a48c9bea5cafbf83ed (table test-table [id=85c5af5ddf7e488c8d0ff1cd5294f28f]) successfully deleted
I20260924 16:06:33.350999 12120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa254cf54d44e2e87bfee33bd404567. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:33.352138 12242 ts_tablet_manager.cc:1918] T 9c4596277b1243a48c9bea5cafbf83ed P 78592a6111ce4606a8f33eed3d2ba83b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:06:33.352380 12242 log.cc:1196] T 9c4596277b1243a48c9bea5cafbf83ed P 78592a6111ce4606a8f33eed3d2ba83b: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-0/wal/wals/9c4596277b1243a48c9bea5cafbf83ed
I20260924 16:06:33.352995 12242 ts_tablet_manager.cc:1939] T 9c4596277b1243a48c9bea5cafbf83ed P 78592a6111ce4606a8f33eed3d2ba83b: Deleting consensus metadata
I20260924 16:06:33.355865 11685 catalog_manager.cc:4841] TS 78592a6111ce4606a8f33eed3d2ba83b (127.3.214.193:42201): tablet 9c4596277b1243a48c9bea5cafbf83ed (table test-table [id=85c5af5ddf7e488c8d0ff1cd5294f28f]) successfully deleted
I20260924 16:06:33.357033 12218 tablet_bootstrap.cc:492] T eaa254cf54d44e2e87bfee33bd404567 P a36b510d73684a5ea8a77d4745925790: Bootstrap starting.
I20260924 16:06:33.358275 12217 tablet_bootstrap.cc:492] T eaa254cf54d44e2e87bfee33bd404567 P 575a2d7c7d554b3fbbe872b0a4601ca1: No bootstrap required, opened a new log
I20260924 16:06:33.358525 12217 ts_tablet_manager.cc:1397] T eaa254cf54d44e2e87bfee33bd404567 P 575a2d7c7d554b3fbbe872b0a4601ca1: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.004s
I20260924 16:06:33.358978 12218 tablet_bootstrap.cc:654] T eaa254cf54d44e2e87bfee33bd404567 P a36b510d73684a5ea8a77d4745925790: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:33.359441 12217 raft_consensus.cc:348] T eaa254cf54d44e2e87bfee33bd404567 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:33.359699 12217 raft_consensus.cc:374] T eaa254cf54d44e2e87bfee33bd404567 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:33.359797 12217 raft_consensus.cc:729] T eaa254cf54d44e2e87bfee33bd404567 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 575a2d7c7d554b3fbbe872b0a4601ca1, State: Initialized, Role: FOLLOWER
I20260924 16:06:33.360128 12217 consensus_queue.cc:260] T eaa254cf54d44e2e87bfee33bd404567 P 575a2d7c7d554b3fbbe872b0a4601ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:33.360821 12217 ts_tablet_manager.cc:1428] T eaa254cf54d44e2e87bfee33bd404567 P 575a2d7c7d554b3fbbe872b0a4601ca1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 16:06:33.362563 12218 tablet_bootstrap.cc:492] T eaa254cf54d44e2e87bfee33bd404567 P a36b510d73684a5ea8a77d4745925790: No bootstrap required, opened a new log
I20260924 16:06:33.362758 12218 ts_tablet_manager.cc:1397] T eaa254cf54d44e2e87bfee33bd404567 P a36b510d73684a5ea8a77d4745925790: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:06:33.363751 12218 raft_consensus.cc:348] T eaa254cf54d44e2e87bfee33bd404567 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:33.364385 12218 raft_consensus.cc:374] T eaa254cf54d44e2e87bfee33bd404567 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:33.364612 12218 raft_consensus.cc:729] T eaa254cf54d44e2e87bfee33bd404567 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a36b510d73684a5ea8a77d4745925790, State: Initialized, Role: FOLLOWER
I20260924 16:06:33.365011 12218 consensus_queue.cc:260] T eaa254cf54d44e2e87bfee33bd404567 P a36b510d73684a5ea8a77d4745925790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:33.365881 12218 ts_tablet_manager.cc:1428] T eaa254cf54d44e2e87bfee33bd404567 P a36b510d73684a5ea8a77d4745925790: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 16:06:33.440348 12225 raft_consensus.cc:482] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:06:33.440589 12225 raft_consensus.cc:504] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:33.442415 12225 leader_election.cc:288] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 575a2d7c7d554b3fbbe872b0a4601ca1 (127.3.214.194:43513), a36b510d73684a5ea8a77d4745925790 (127.3.214.195:37245)
I20260924 16:06:33.451360 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.443080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:45477 (local address 127.3.214.194:43513)
0924 16:06:33.443311 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.443316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.444532 (+ 1216us) server_negotiation.cc:407] Connection header received
0924 16:06:33.444751 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.444757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.444838 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.444937 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:33.445945 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.447022 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.449079 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.450087 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.450318 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.450402 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:33.450852 (+ 450us) server_negotiation.cc:299] Negotiation successful
0924 16:06:33.451007 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 16:06:33.451471 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.443602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:43513 (local address 127.3.214.193:45477)
0924 16:06:33.444440 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:33.444458 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:33.444612 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:33.445113 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:33.445118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:33.445159 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:33.445744 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.445759 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.447166 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.447173 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.448897 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.448914 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.450390 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.450397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.450619 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.450682 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 16:06:33.450796 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 16:06:33.450990 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":461,"threads_started":1}
I20260924 16:06:33.452366 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.443112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:44729 (local address 127.3.214.195:37245)
0924 16:06:33.443301 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.443308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.444158 (+ 850us) server_negotiation.cc:407] Connection header received
0924 16:06:33.444585 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.444593 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.444717 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.444865 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:06:33.445561 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.446755 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.449640 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.450857 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.451021 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.451093 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:33.451942 (+ 849us) server_negotiation.cc:299] Negotiation successful
0924 16:06:33.452090 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:06:33.452404 12000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa254cf54d44e2e87bfee33bd404567" candidate_uuid: "78592a6111ce4606a8f33eed3d2ba83b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" is_pre_election: true
I20260924 16:06:33.452767 12000 raft_consensus.cc:2454] T eaa254cf54d44e2e87bfee33bd404567 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78592a6111ce4606a8f33eed3d2ba83b in term 0.
I20260924 16:06:33.453140 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.443144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:37245 (local address 127.3.214.193:44729)
0924 16:06:33.444026 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:33.444048 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:33.444304 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:33.444898 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:33.444903 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:33.444929 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:06:33.445348 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.445363 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.446928 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.446934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.449421 (+ 2487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.449442 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.451314 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.451321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.451712 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.451752 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 16:06:33.452634 (+ 882us) client_negotiation.cc:240] Negotiation successful
0924 16:06:33.452791 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":540,"threads_started":1}
I20260924 16:06:33.453471 11788 leader_election.cc:302] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 575a2d7c7d554b3fbbe872b0a4601ca1, 78592a6111ce4606a8f33eed3d2ba83b; no voters:
I20260924 16:06:33.454169 12225 raft_consensus.cc:2781] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:06:33.454329 12225 raft_consensus.cc:482] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:06:33.454439 12225 raft_consensus.cc:3037] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:33.454245 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa254cf54d44e2e87bfee33bd404567" candidate_uuid: "78592a6111ce4606a8f33eed3d2ba83b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a36b510d73684a5ea8a77d4745925790" is_pre_election: true
I20260924 16:06:33.454823 12146 raft_consensus.cc:2454] T eaa254cf54d44e2e87bfee33bd404567 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78592a6111ce4606a8f33eed3d2ba83b in term 0.
I20260924 16:06:33.457170 12225 raft_consensus.cc:504] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:33.457834 12225 leader_election.cc:288] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b [CANDIDATE]: Term 1 election: Requested vote from peers 575a2d7c7d554b3fbbe872b0a4601ca1 (127.3.214.194:43513), a36b510d73684a5ea8a77d4745925790 (127.3.214.195:37245)
I20260924 16:06:33.458739 12000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa254cf54d44e2e87bfee33bd404567" candidate_uuid: "78592a6111ce4606a8f33eed3d2ba83b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1"
I20260924 16:06:33.459105 12000 raft_consensus.cc:3037] T eaa254cf54d44e2e87bfee33bd404567 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:33.461037 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa254cf54d44e2e87bfee33bd404567" candidate_uuid: "78592a6111ce4606a8f33eed3d2ba83b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a36b510d73684a5ea8a77d4745925790"
I20260924 16:06:33.461485 12146 raft_consensus.cc:3037] T eaa254cf54d44e2e87bfee33bd404567 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:33.462833 12000 raft_consensus.cc:2454] T eaa254cf54d44e2e87bfee33bd404567 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78592a6111ce4606a8f33eed3d2ba83b in term 1.
I20260924 16:06:33.463544 11788 leader_election.cc:302] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 575a2d7c7d554b3fbbe872b0a4601ca1, 78592a6111ce4606a8f33eed3d2ba83b; no voters:
I20260924 16:06:33.464017 12225 raft_consensus.cc:2781] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:33.464811 12146 raft_consensus.cc:2454] T eaa254cf54d44e2e87bfee33bd404567 P a36b510d73684a5ea8a77d4745925790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78592a6111ce4606a8f33eed3d2ba83b in term 1.
I20260924 16:06:33.465401 12225 raft_consensus.cc:686] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b [term 1 LEADER]: Becoming Leader. State: Replica: 78592a6111ce4606a8f33eed3d2ba83b, State: Running, Role: LEADER
I20260924 16:06:33.465893 12225 consensus_queue.cc:237] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:33.470516 11700 catalog_manager.cc:5495] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b reported cstate change: term changed from 0 to 1, leader changed from <none> to 78592a6111ce4606a8f33eed3d2ba83b (127.3.214.193). New cstate: current_term: 1 leader_uuid: "78592a6111ce4606a8f33eed3d2ba83b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } health_report { overall_health: UNKNOWN } } }
Sep 24 16:06:33 dist-test-slave-1gc3 krb5kdc[11655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265989, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.254@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM
I20260924 16:06:33.537151 12258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.521308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39763 (local address 127.0.0.1:40132)
0924 16:06:33.522261 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:33.522280 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:33.522456 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:33.522822 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:33.522827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:33.523662 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:33.523919 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.523929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.525040 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.525043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.526064 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.526077 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.526852 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.526857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.527022 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.527868 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:33.527892 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:33.530735 (+ 2843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:33.534409 (+ 3674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:33.534450 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:33.534462 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:33.534958 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:33.535318 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:33.535323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:33.535327 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:33.535472 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:33.536019 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:33.536029 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:33.536210 (+ 181us) client_negotiation.cc:769] Sending connection context
0924 16:06:33.536358 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 16:06:33.536677 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":606,"threads_started":1}
I20260924 16:06:33.537197 12257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.521083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40132 (local address 127.3.214.254:39763)
0924 16:06:33.522074 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.522084 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.522388 (+ 304us) server_negotiation.cc:407] Connection header received
0924 16:06:33.522589 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.522593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.522667 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.522757 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.524107 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.524893 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.526237 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.526682 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.526799 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.530997 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:33.531002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:33.531006 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:33.531037 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:33.534236 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.535093 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.535097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.535099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.535216 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:33.535580 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:33.535583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:33.535585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:33.535884 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:33.536027 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:33.536482 (+ 455us) server_negotiation.cc:299] Negotiation successful
0924 16:06:33.536839 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":788,"thread_start_us":513,"threads_started":1}
I20260924 16:06:33.544539 11700 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:40132:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 16:06:33.545742 11700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:06:33.554083 11980 tablet_service.cc:1467] Processing CreateTablet for tablet cd902cfdd4894036a148e5b84fd4bdc3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=66109ea009694350973c9d0ce38be560]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:06:33.554229 11838 tablet_service.cc:1467] Processing CreateTablet for tablet cd902cfdd4894036a148e5b84fd4bdc3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=66109ea009694350973c9d0ce38be560]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:06:33.554726 11980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd902cfdd4894036a148e5b84fd4bdc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:33.554965 11838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd902cfdd4894036a148e5b84fd4bdc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:33.559990 12120 tablet_service.cc:1467] Processing CreateTablet for tablet cd902cfdd4894036a148e5b84fd4bdc3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=66109ea009694350973c9d0ce38be560]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:06:33.560652 12120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd902cfdd4894036a148e5b84fd4bdc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:33.561038 12217 tablet_bootstrap.cc:492] T cd902cfdd4894036a148e5b84fd4bdc3 P 575a2d7c7d554b3fbbe872b0a4601ca1: Bootstrap starting.
I20260924 16:06:33.562851 12217 tablet_bootstrap.cc:654] T cd902cfdd4894036a148e5b84fd4bdc3 P 575a2d7c7d554b3fbbe872b0a4601ca1: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:33.563277 12221 tablet_bootstrap.cc:492] T cd902cfdd4894036a148e5b84fd4bdc3 P 78592a6111ce4606a8f33eed3d2ba83b: Bootstrap starting.
I20260924 16:06:33.565449 12221 tablet_bootstrap.cc:654] T cd902cfdd4894036a148e5b84fd4bdc3 P 78592a6111ce4606a8f33eed3d2ba83b: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:33.570052 12218 tablet_bootstrap.cc:492] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790: Bootstrap starting.
I20260924 16:06:33.572141 12218 tablet_bootstrap.cc:654] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:33.576840 12221 tablet_bootstrap.cc:492] T cd902cfdd4894036a148e5b84fd4bdc3 P 78592a6111ce4606a8f33eed3d2ba83b: No bootstrap required, opened a new log
I20260924 16:06:33.577188 12221 ts_tablet_manager.cc:1397] T cd902cfdd4894036a148e5b84fd4bdc3 P 78592a6111ce4606a8f33eed3d2ba83b: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
I20260924 16:06:33.577698 12218 tablet_bootstrap.cc:492] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790: No bootstrap required, opened a new log
I20260924 16:06:33.577983 12218 ts_tablet_manager.cc:1397] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 16:06:33.578132 12221 raft_consensus.cc:348] T cd902cfdd4894036a148e5b84fd4bdc3 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:33.578421 12221 raft_consensus.cc:374] T cd902cfdd4894036a148e5b84fd4bdc3 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:33.578508 12221 raft_consensus.cc:729] T cd902cfdd4894036a148e5b84fd4bdc3 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78592a6111ce4606a8f33eed3d2ba83b, State: Initialized, Role: FOLLOWER
I20260924 16:06:33.578845 12221 consensus_queue.cc:260] T cd902cfdd4894036a148e5b84fd4bdc3 P 78592a6111ce4606a8f33eed3d2ba83b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:33.579303 12217 tablet_bootstrap.cc:492] T cd902cfdd4894036a148e5b84fd4bdc3 P 575a2d7c7d554b3fbbe872b0a4601ca1: No bootstrap required, opened a new log
I20260924 16:06:33.579526 12217 ts_tablet_manager.cc:1397] T cd902cfdd4894036a148e5b84fd4bdc3 P 575a2d7c7d554b3fbbe872b0a4601ca1: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.009s
I20260924 16:06:33.579356 12225 tablet_replica.cc:439] T cd902cfdd4894036a148e5b84fd4bdc3 P 78592a6111ce4606a8f33eed3d2ba83b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } } }
I20260924 16:06:33.579694 12225 tablet_replica.cc:442] T cd902cfdd4894036a148e5b84fd4bdc3 P 78592a6111ce4606a8f33eed3d2ba83b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:06:33.580116 12218 raft_consensus.cc:348] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:33.580479 12218 raft_consensus.cc:374] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:33.580415 12217 raft_consensus.cc:348] T cd902cfdd4894036a148e5b84fd4bdc3 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:33.580681 12218 raft_consensus.cc:729] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a36b510d73684a5ea8a77d4745925790, State: Initialized, Role: FOLLOWER
I20260924 16:06:33.580687 12217 raft_consensus.cc:374] T cd902cfdd4894036a148e5b84fd4bdc3 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:33.580771 12217 raft_consensus.cc:729] T cd902cfdd4894036a148e5b84fd4bdc3 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 575a2d7c7d554b3fbbe872b0a4601ca1, State: Initialized, Role: FOLLOWER
I20260924 16:06:33.581045 12218 consensus_queue.cc:260] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:33.581046 12217 consensus_queue.cc:260] T cd902cfdd4894036a148e5b84fd4bdc3 P 575a2d7c7d554b3fbbe872b0a4601ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:33.581701 12217 ts_tablet_manager.cc:1428] T cd902cfdd4894036a148e5b84fd4bdc3 P 575a2d7c7d554b3fbbe872b0a4601ca1: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260924 16:06:33.581609 12236 tablet_replica.cc:439] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } } }
I20260924 16:06:33.581984 12236 tablet_replica.cc:442] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:06:33.582311 12223 tablet_replica.cc:439] T cd902cfdd4894036a148e5b84fd4bdc3 P 575a2d7c7d554b3fbbe872b0a4601ca1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } } }
I20260924 16:06:33.582734 12223 tablet_replica.cc:442] T cd902cfdd4894036a148e5b84fd4bdc3 P 575a2d7c7d554b3fbbe872b0a4601ca1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:06:33.583269 12218 ts_tablet_manager.cc:1428] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 16:06:33.584367 12221 ts_tablet_manager.cc:1428] T cd902cfdd4894036a148e5b84fd4bdc3 P 78592a6111ce4606a8f33eed3d2ba83b: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20260924 16:06:33.705092 12236 raft_consensus.cc:482] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:06:33.705379 12236 raft_consensus.cc:504] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:33.706311 12236 leader_election.cc:288] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78592a6111ce4606a8f33eed3d2ba83b (127.3.214.193:42201), 575a2d7c7d554b3fbbe872b0a4601ca1 (127.3.214.194:43513)
I20260924 16:06:33.707165 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd902cfdd4894036a148e5b84fd4bdc3" candidate_uuid: "a36b510d73684a5ea8a77d4745925790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78592a6111ce4606a8f33eed3d2ba83b" is_pre_election: true
I20260924 16:06:33.707188 12000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd902cfdd4894036a148e5b84fd4bdc3" candidate_uuid: "a36b510d73684a5ea8a77d4745925790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" is_pre_election: true
I20260924 16:06:33.707484 11863 raft_consensus.cc:2454] T cd902cfdd4894036a148e5b84fd4bdc3 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a36b510d73684a5ea8a77d4745925790 in term 0.
I20260924 16:06:33.707537 12000 raft_consensus.cc:2454] T cd902cfdd4894036a148e5b84fd4bdc3 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a36b510d73684a5ea8a77d4745925790 in term 0.
I20260924 16:06:33.708093 12070 leader_election.cc:302] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 575a2d7c7d554b3fbbe872b0a4601ca1, a36b510d73684a5ea8a77d4745925790; no voters:
I20260924 16:06:33.708472 12236 raft_consensus.cc:2781] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:06:33.708674 12236 raft_consensus.cc:482] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:06:33.708793 12236 raft_consensus.cc:3037] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:33.711544 12236 raft_consensus.cc:504] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:33.712280 12236 leader_election.cc:288] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790 [CANDIDATE]: Term 1 election: Requested vote from peers 78592a6111ce4606a8f33eed3d2ba83b (127.3.214.193:42201), 575a2d7c7d554b3fbbe872b0a4601ca1 (127.3.214.194:43513)
I20260924 16:06:33.713166 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd902cfdd4894036a148e5b84fd4bdc3" candidate_uuid: "a36b510d73684a5ea8a77d4745925790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78592a6111ce4606a8f33eed3d2ba83b"
I20260924 16:06:33.713199 12000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd902cfdd4894036a148e5b84fd4bdc3" candidate_uuid: "a36b510d73684a5ea8a77d4745925790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1"
I20260924 16:06:33.713438 11863 raft_consensus.cc:3037] T cd902cfdd4894036a148e5b84fd4bdc3 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:33.713565 12000 raft_consensus.cc:3037] T cd902cfdd4894036a148e5b84fd4bdc3 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:33.716058 11863 raft_consensus.cc:2454] T cd902cfdd4894036a148e5b84fd4bdc3 P 78592a6111ce4606a8f33eed3d2ba83b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36b510d73684a5ea8a77d4745925790 in term 1.
I20260924 16:06:33.716691 12070 leader_election.cc:302] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78592a6111ce4606a8f33eed3d2ba83b, a36b510d73684a5ea8a77d4745925790; no voters:
I20260924 16:06:33.716885 12000 raft_consensus.cc:2454] T cd902cfdd4894036a148e5b84fd4bdc3 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36b510d73684a5ea8a77d4745925790 in term 1.
I20260924 16:06:33.717054 12236 raft_consensus.cc:2781] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:33.717262 12236 raft_consensus.cc:686] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790 [term 1 LEADER]: Becoming Leader. State: Replica: a36b510d73684a5ea8a77d4745925790, State: Running, Role: LEADER
I20260924 16:06:33.717643 12236 consensus_queue.cc:237] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:33.721010 12224 tablet_replica.cc:439] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790: TxnStatusTablet state changed. Reason: New leader a36b510d73684a5ea8a77d4745925790. Latest consensus state: current_term: 1 leader_uuid: "a36b510d73684a5ea8a77d4745925790" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } } }
I20260924 16:06:33.721359 12224 tablet_replica.cc:442] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:06:33.721731 11701 catalog_manager.cc:5495] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790 reported cstate change: term changed from 0 to 1, leader changed from <none> to a36b510d73684a5ea8a77d4745925790 (127.3.214.195). New cstate: current_term: 1 leader_uuid: "a36b510d73684a5ea8a77d4745925790" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } health_report { overall_health: HEALTHY } } }
I20260924 16:06:33.723042 12263 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 16:06:33.723414 12263 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 16:06:33.744010 12258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.737299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:37245 (local address 127.0.0.1:55876)
0924 16:06:33.737632 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:33.737654 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:33.737811 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:33.738503 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:33.738509 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:33.738522 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:06:33.738726 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.738736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.740458 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.740462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.741565 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.741581 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.742697 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.742702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.742882 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.742962 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:06:33.743474 (+ 512us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:06:33.743492 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 16:06:33.743568 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 16:06:33.743717 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 16:06:33.744087 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.737271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55876 (local address 127.3.214.195:37245)
0924 16:06:33.738162 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.738173 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.738193 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:06:33.738271 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.738275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.738358 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.738449 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:33.738997 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.740263 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.741755 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.742492 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.742676 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.743183 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:33.743347 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:33.743480 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:33.743632 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 16:06:33.743740 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670}
I20260924 16:06:33.748705 12000 raft_consensus.cc:1261] T cd902cfdd4894036a148e5b84fd4bdc3 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 1 FOLLOWER]: Refusing update from remote peer a36b510d73684a5ea8a77d4745925790: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:33.748695 11863 raft_consensus.cc:1261] T cd902cfdd4894036a148e5b84fd4bdc3 P 78592a6111ce4606a8f33eed3d2ba83b [term 1 FOLLOWER]: Refusing update from remote peer a36b510d73684a5ea8a77d4745925790: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:33.749954 12224 consensus_queue.cc:1035] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790 [LEADER]: Connected to new peer: Peer: permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:33.750424 12236 consensus_queue.cc:1035] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:33.755241 12223 tablet_replica.cc:439] T cd902cfdd4894036a148e5b84fd4bdc3 P 575a2d7c7d554b3fbbe872b0a4601ca1: TxnStatusTablet state changed. Reason: New leader a36b510d73684a5ea8a77d4745925790. Latest consensus state: current_term: 1 leader_uuid: "a36b510d73684a5ea8a77d4745925790" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } } }
I20260924 16:06:33.755605 12223 tablet_replica.cc:442] T cd902cfdd4894036a148e5b84fd4bdc3 P 575a2d7c7d554b3fbbe872b0a4601ca1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:06:33.757506 12225 tablet_replica.cc:439] T cd902cfdd4894036a148e5b84fd4bdc3 P 78592a6111ce4606a8f33eed3d2ba83b: TxnStatusTablet state changed. Reason: New leader a36b510d73684a5ea8a77d4745925790. Latest consensus state: current_term: 1 leader_uuid: "a36b510d73684a5ea8a77d4745925790" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } } }
I20260924 16:06:33.757887 12225 tablet_replica.cc:442] T cd902cfdd4894036a148e5b84fd4bdc3 P 78592a6111ce4606a8f33eed3d2ba83b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:06:33.759773 12228 tablet_replica.cc:439] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a36b510d73684a5ea8a77d4745925790" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } } }
I20260924 16:06:33.760171 12228 tablet_replica.cc:442] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:06:33.760432 12228 tablet_replica.cc:439] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a36b510d73684a5ea8a77d4745925790" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } } }
I20260924 16:06:33.760663 12228 tablet_replica.cc:442] T cd902cfdd4894036a148e5b84fd4bdc3 P a36b510d73684a5ea8a77d4745925790: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:06:33.760788 12223 tablet_replica.cc:439] T cd902cfdd4894036a148e5b84fd4bdc3 P 575a2d7c7d554b3fbbe872b0a4601ca1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a36b510d73684a5ea8a77d4745925790" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } } }
I20260924 16:06:33.761094 12223 tablet_replica.cc:442] T cd902cfdd4894036a148e5b84fd4bdc3 P 575a2d7c7d554b3fbbe872b0a4601ca1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:06:33.761652 12233 mvcc.cc:205] Tried to move back new op lower bound from 7332929510387335168 to 7332929510270676992. Current Snapshot: MvccSnapshot[applied={T|T < 7332929510387335168}]
I20260924 16:06:33.764221 12231 mvcc.cc:205] Tried to move back new op lower bound from 7332929510387335168 to 7332929510270676992. Current Snapshot: MvccSnapshot[applied={T|T < 7332929510387335168 or (T in {7332929510387335168})}]
I20260924 16:06:33.765241 12225 tablet_replica.cc:439] T cd902cfdd4894036a148e5b84fd4bdc3 P 78592a6111ce4606a8f33eed3d2ba83b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a36b510d73684a5ea8a77d4745925790" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } } }
I20260924 16:06:33.765589 12225 tablet_replica.cc:442] T cd902cfdd4894036a148e5b84fd4bdc3 P 78592a6111ce4606a8f33eed3d2ba83b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:06:33.780311 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.770900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44040 (local address 127.3.214.193:42201)
0924 16:06:33.772539 (+ 1639us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.772544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.772567 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:06:33.772695 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.772699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.772799 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.772974 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:33.773727 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.774780 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.777028 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.777666 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.777790 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.778491 (+ 701us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:33.779355 (+ 864us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:33.779489 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:33.779726 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 16:06:33.779840 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1518,"thread_start_us":1345,"threads_started":1}
I20260924 16:06:33.803014 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.795379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:36483 (local address 127.3.214.195:37245)
0924 16:06:33.795666 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.795674 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.796471 (+ 797us) server_negotiation.cc:407] Connection header received
0924 16:06:33.796725 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.796730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.796819 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.796915 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:33.797582 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.798606 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.800845 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.801394 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.801521 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.802168 (+ 647us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:33.802377 (+ 209us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:33.802452 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:33.802614 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 16:06:33.802709 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 16:06:33.803402 12272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.795372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:37245 (local address 127.3.214.193:36483)
0924 16:06:33.796331 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:33.796356 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:33.796561 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:33.796959 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:33.796964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:33.796978 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:06:33.797389 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.797403 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.798796 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.798802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.800599 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.800622 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.801627 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.801634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.801830 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.801930 (+ 100us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:06:33.802480 (+ 550us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:06:33.802501 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 16:06:33.802588 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 16:06:33.802753 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":439,"threads_started":1}
I20260924 16:06:33.816263 12000 raft_consensus.cc:1261] T eaa254cf54d44e2e87bfee33bd404567 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 1 FOLLOWER]: Refusing update from remote peer 78592a6111ce4606a8f33eed3d2ba83b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:33.816471 12146 raft_consensus.cc:1261] T eaa254cf54d44e2e87bfee33bd404567 P a36b510d73684a5ea8a77d4745925790 [term 1 FOLLOWER]: Refusing update from remote peer 78592a6111ce4606a8f33eed3d2ba83b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:33.817237 12249 consensus_queue.cc:1035] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b [LEADER]: Connected to new peer: Peer: permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 16:06:33.818005 12225 consensus_queue.cc:1035] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:33.825009 12232 mvcc.cc:205] Tried to move back new op lower bound from 7332929510661496832 to 7332929509240877056. Current Snapshot: MvccSnapshot[applied={T|T < 7332929510661496832}]
I20260924 16:06:33.866401 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.858187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:54587 (local address 127.3.214.193:42201)
0924 16:06:33.858399 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.858406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.858878 (+ 472us) server_negotiation.cc:407] Connection header received
0924 16:06:33.859220 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.859227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.859331 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.859399 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:33.860404 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.861573 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.863610 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.864223 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.864372 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.865371 (+ 999us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:33.865531 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:33.865619 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:33.865861 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 16:06:33.865975 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:06:33.866906 12276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:33.858010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:42201 (local address 127.3.214.195:54587)
0924 16:06:33.858764 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:33.858787 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:33.859017 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:33.859644 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:33.859649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:33.859662 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:06:33.860163 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.860179 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.861787 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.861792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.863409 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:33.863428 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.864607 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.864623 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:33.864933 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.865053 (+ 120us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:06:33.865690 (+ 637us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:06:33.865720 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 16:06:33.865805 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 16:06:33.865996 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":361,"threads_started":1}
I20260924 16:06:33.913470 11701 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:33.913865 11701 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:33.918872 11701 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a: Sending DeleteTablet for 3 replicas of tablet eaa254cf54d44e2e87bfee33bd404567
I20260924 16:06:33.920320 11980 tablet_service.cc:1514] Processing DeleteTablet for tablet eaa254cf54d44e2e87bfee33bd404567 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:33 UTC) from {username='oryx', principal='oryx/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:60260
I20260924 16:06:33.920642 12120 tablet_service.cc:1514] Processing DeleteTablet for tablet eaa254cf54d44e2e87bfee33bd404567 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:33 UTC) from {username='oryx', principal='oryx/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:55858
I20260924 16:06:33.921389 11838 tablet_service.cc:1514] Processing DeleteTablet for tablet eaa254cf54d44e2e87bfee33bd404567 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:33 UTC) from {username='oryx', principal='oryx/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:44034
I20260924 16:06:33.922470 12281 tablet_replica.cc:330] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b: stopping tablet replica
I20260924 16:06:33.922801 12281 raft_consensus.cc:2229] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:06:33.923626 12281 raft_consensus.cc:2258] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:33.924937 12281 ts_tablet_manager.cc:1905] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:33.933882 12281 ts_tablet_manager.cc:1918] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:06:33.934293 12281 log.cc:1196] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-0/wal/wals/eaa254cf54d44e2e87bfee33bd404567
I20260924 16:06:33.935009 12281 ts_tablet_manager.cc:1939] T eaa254cf54d44e2e87bfee33bd404567 P 78592a6111ce4606a8f33eed3d2ba83b: Deleting consensus metadata
I20260924 16:06:33.936498 11685 catalog_manager.cc:4841] TS 78592a6111ce4606a8f33eed3d2ba83b (127.3.214.193:42201): tablet eaa254cf54d44e2e87bfee33bd404567 (table test-table [id=3593fd570f144f0b95e6888f79dd2487]) successfully deleted
I20260924 16:06:33.940964 12280 tablet_replica.cc:330] T eaa254cf54d44e2e87bfee33bd404567 P 575a2d7c7d554b3fbbe872b0a4601ca1: stopping tablet replica
I20260924 16:06:33.941398 12280 raft_consensus.cc:2229] T eaa254cf54d44e2e87bfee33bd404567 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:06:33.942134 12282 tablet_replica.cc:330] T eaa254cf54d44e2e87bfee33bd404567 P a36b510d73684a5ea8a77d4745925790: stopping tablet replica
I20260924 16:06:33.942451 12280 raft_consensus.cc:2258] T eaa254cf54d44e2e87bfee33bd404567 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:33.943723 12280 ts_tablet_manager.cc:1905] T eaa254cf54d44e2e87bfee33bd404567 P 575a2d7c7d554b3fbbe872b0a4601ca1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:33.951148 12280 ts_tablet_manager.cc:1918] T eaa254cf54d44e2e87bfee33bd404567 P 575a2d7c7d554b3fbbe872b0a4601ca1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:06:33.951416 12280 log.cc:1196] T eaa254cf54d44e2e87bfee33bd404567 P 575a2d7c7d554b3fbbe872b0a4601ca1: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/eaa254cf54d44e2e87bfee33bd404567
I20260924 16:06:33.951923 12280 ts_tablet_manager.cc:1939] T eaa254cf54d44e2e87bfee33bd404567 P 575a2d7c7d554b3fbbe872b0a4601ca1: Deleting consensus metadata
I20260924 16:06:33.953780 11685 catalog_manager.cc:4841] TS 575a2d7c7d554b3fbbe872b0a4601ca1 (127.3.214.194:43513): tablet eaa254cf54d44e2e87bfee33bd404567 (table test-table [id=3593fd570f144f0b95e6888f79dd2487]) successfully deleted
I20260924 16:06:33.954327 12282 raft_consensus.cc:2229] T eaa254cf54d44e2e87bfee33bd404567 P a36b510d73684a5ea8a77d4745925790 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:06:33.955344 12282 raft_consensus.cc:2258] T eaa254cf54d44e2e87bfee33bd404567 P a36b510d73684a5ea8a77d4745925790 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:33.956828 12282 ts_tablet_manager.cc:1905] T eaa254cf54d44e2e87bfee33bd404567 P a36b510d73684a5ea8a77d4745925790: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:33.965029 12282 ts_tablet_manager.cc:1918] T eaa254cf54d44e2e87bfee33bd404567 P a36b510d73684a5ea8a77d4745925790: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:06:33.965296 12282 log.cc:1196] T eaa254cf54d44e2e87bfee33bd404567 P a36b510d73684a5ea8a77d4745925790: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/eaa254cf54d44e2e87bfee33bd404567
I20260924 16:06:33.965942 12282 ts_tablet_manager.cc:1939] T eaa254cf54d44e2e87bfee33bd404567 P a36b510d73684a5ea8a77d4745925790: Deleting consensus metadata
I20260924 16:06:33.967592 11688 catalog_manager.cc:4841] TS a36b510d73684a5ea8a77d4745925790 (127.3.214.195:37245): tablet eaa254cf54d44e2e87bfee33bd404567 (table test-table [id=3593fd570f144f0b95e6888f79dd2487]) successfully deleted
Sep 24 16:06:33 dist-test-slave-1gc3 krb5kdc[11655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 16:06:33 dist-test-slave-1gc3 krb5kdc[11655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:33 dist-test-slave-1gc3 krb5kdc[11655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 16:06:33.998871 12257 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:33.987968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40136 (local address 127.3.214.254:39763)
0924 16:06:33.988208 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:33.988215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:33.988714 (+ 499us) server_negotiation.cc:407] Connection header received
0924 16:06:33.988906 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:33.988911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:33.988976 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:33.989060 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:33.990490 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.991252 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.992486 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:33.992915 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:33.993025 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:33.998608 (+ 5583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40136: BlockingRecv error: recv got EOF from 127.0.0.1:40136 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 16:06:34 dist-test-slave-1gc3 krb5kdc[11655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM
I20260924 16:06:34.016798 12257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:34.002871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40138 (local address 127.3.214.254:39763)
0924 16:06:34.003069 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:34.003076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:34.003502 (+ 426us) server_negotiation.cc:407] Connection header received
0924 16:06:34.003674 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:34.003679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:34.003751 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:34.003828 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:34.005045 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.005817 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.007174 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.007592 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.007704 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:34.011525 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:34.011530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:34.011533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:34.011574 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:34.014158 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:34.014921 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:34.014927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:34.014930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:34.015076 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:34.015438 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:34.015442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:34.015444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:34.015735 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:34.016002 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:34.016330 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 16:06:34.016534 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 16:06:34.021206 11701 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:06:34.021945 11701 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:06:34.032711 11838 tablet_service.cc:1467] Processing CreateTablet for tablet f5e1113468cd434d9a848d79f9e610e8 (DEFAULT_TABLE table=test-table [id=b7f7e94df39146948a0a17275c2238a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:34.032717 12120 tablet_service.cc:1467] Processing CreateTablet for tablet f5e1113468cd434d9a848d79f9e610e8 (DEFAULT_TABLE table=test-table [id=b7f7e94df39146948a0a17275c2238a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:34.033227 11838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e1113468cd434d9a848d79f9e610e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:34.033221 12120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e1113468cd434d9a848d79f9e610e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:34.040580 11980 tablet_service.cc:1467] Processing CreateTablet for tablet f5e1113468cd434d9a848d79f9e610e8 (DEFAULT_TABLE table=test-table [id=b7f7e94df39146948a0a17275c2238a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:34.040823 12221 tablet_bootstrap.cc:492] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b: Bootstrap starting.
I20260924 16:06:34.041064 11980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e1113468cd434d9a848d79f9e610e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:34.042567 12221 tablet_bootstrap.cc:654] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:34.050030 12217 tablet_bootstrap.cc:492] T f5e1113468cd434d9a848d79f9e610e8 P 575a2d7c7d554b3fbbe872b0a4601ca1: Bootstrap starting.
I20260924 16:06:34.051331 12218 tablet_bootstrap.cc:492] T f5e1113468cd434d9a848d79f9e610e8 P a36b510d73684a5ea8a77d4745925790: Bootstrap starting.
I20260924 16:06:34.052181 12217 tablet_bootstrap.cc:654] T f5e1113468cd434d9a848d79f9e610e8 P 575a2d7c7d554b3fbbe872b0a4601ca1: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:34.053515 12218 tablet_bootstrap.cc:654] T f5e1113468cd434d9a848d79f9e610e8 P a36b510d73684a5ea8a77d4745925790: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:34.058022 12218 tablet_bootstrap.cc:492] T f5e1113468cd434d9a848d79f9e610e8 P a36b510d73684a5ea8a77d4745925790: No bootstrap required, opened a new log
I20260924 16:06:34.058238 12218 ts_tablet_manager.cc:1397] T f5e1113468cd434d9a848d79f9e610e8 P a36b510d73684a5ea8a77d4745925790: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 16:06:34.059180 12218 raft_consensus.cc:348] T f5e1113468cd434d9a848d79f9e610e8 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:34.059508 12218 raft_consensus.cc:374] T f5e1113468cd434d9a848d79f9e610e8 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:34.059615 12218 raft_consensus.cc:729] T f5e1113468cd434d9a848d79f9e610e8 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a36b510d73684a5ea8a77d4745925790, State: Initialized, Role: FOLLOWER
I20260924 16:06:34.059901 12218 consensus_queue.cc:260] T f5e1113468cd434d9a848d79f9e610e8 P a36b510d73684a5ea8a77d4745925790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:34.061054 12218 ts_tablet_manager.cc:1428] T f5e1113468cd434d9a848d79f9e610e8 P a36b510d73684a5ea8a77d4745925790: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 16:06:34.063581 12221 tablet_bootstrap.cc:492] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b: No bootstrap required, opened a new log
I20260924 16:06:34.063750 12221 ts_tablet_manager.cc:1397] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.008s
I20260924 16:06:34.064736 12221 raft_consensus.cc:348] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:34.065027 12221 raft_consensus.cc:374] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:34.065125 12221 raft_consensus.cc:729] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78592a6111ce4606a8f33eed3d2ba83b, State: Initialized, Role: FOLLOWER
I20260924 16:06:34.065491 12221 consensus_queue.cc:260] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:34.066085 12221 ts_tablet_manager.cc:1428] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260924 16:06:34.075332 12217 tablet_bootstrap.cc:492] T f5e1113468cd434d9a848d79f9e610e8 P 575a2d7c7d554b3fbbe872b0a4601ca1: No bootstrap required, opened a new log
I20260924 16:06:34.075575 12217 ts_tablet_manager.cc:1397] T f5e1113468cd434d9a848d79f9e610e8 P 575a2d7c7d554b3fbbe872b0a4601ca1: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.005s
I20260924 16:06:34.076586 12217 raft_consensus.cc:348] T f5e1113468cd434d9a848d79f9e610e8 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:34.076936 12217 raft_consensus.cc:374] T f5e1113468cd434d9a848d79f9e610e8 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:34.077044 12217 raft_consensus.cc:729] T f5e1113468cd434d9a848d79f9e610e8 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 575a2d7c7d554b3fbbe872b0a4601ca1, State: Initialized, Role: FOLLOWER
I20260924 16:06:34.077301 12217 consensus_queue.cc:260] T f5e1113468cd434d9a848d79f9e610e8 P 575a2d7c7d554b3fbbe872b0a4601ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:34.080031 12217 ts_tablet_manager.cc:1428] T f5e1113468cd434d9a848d79f9e610e8 P 575a2d7c7d554b3fbbe872b0a4601ca1: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260924 16:06:34.090812 12278 raft_consensus.cc:482] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:06:34.091168 12278 raft_consensus.cc:504] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:34.092252 12278 leader_election.cc:288] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 575a2d7c7d554b3fbbe872b0a4601ca1 (127.3.214.194:43513), a36b510d73684a5ea8a77d4745925790 (127.3.214.195:37245)
I20260924 16:06:34.093436 11999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e1113468cd434d9a848d79f9e610e8" candidate_uuid: "78592a6111ce4606a8f33eed3d2ba83b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" is_pre_election: true
I20260924 16:06:34.093765 11999 raft_consensus.cc:2454] T f5e1113468cd434d9a848d79f9e610e8 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78592a6111ce4606a8f33eed3d2ba83b in term 0.
I20260924 16:06:34.094694 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e1113468cd434d9a848d79f9e610e8" candidate_uuid: "78592a6111ce4606a8f33eed3d2ba83b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a36b510d73684a5ea8a77d4745925790" is_pre_election: true
I20260924 16:06:34.095016 12146 raft_consensus.cc:2454] T f5e1113468cd434d9a848d79f9e610e8 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78592a6111ce4606a8f33eed3d2ba83b in term 0.
I20260924 16:06:34.095708 11788 leader_election.cc:302] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 575a2d7c7d554b3fbbe872b0a4601ca1, 78592a6111ce4606a8f33eed3d2ba83b; no voters:
I20260924 16:06:34.096247 12278 raft_consensus.cc:2781] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:06:34.096441 12278 raft_consensus.cc:482] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:06:34.096542 12278 raft_consensus.cc:3037] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:34.100095 12278 raft_consensus.cc:504] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:34.101593 12278 leader_election.cc:288] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b [CANDIDATE]: Term 1 election: Requested vote from peers 575a2d7c7d554b3fbbe872b0a4601ca1 (127.3.214.194:43513), a36b510d73684a5ea8a77d4745925790 (127.3.214.195:37245)
I20260924 16:06:34.101892 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e1113468cd434d9a848d79f9e610e8" candidate_uuid: "78592a6111ce4606a8f33eed3d2ba83b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a36b510d73684a5ea8a77d4745925790"
I20260924 16:06:34.102263 12146 raft_consensus.cc:3037] T f5e1113468cd434d9a848d79f9e610e8 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:34.102993 11999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e1113468cd434d9a848d79f9e610e8" candidate_uuid: "78592a6111ce4606a8f33eed3d2ba83b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1"
I20260924 16:06:34.103278 11999 raft_consensus.cc:3037] T f5e1113468cd434d9a848d79f9e610e8 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:34.105975 12146 raft_consensus.cc:2454] T f5e1113468cd434d9a848d79f9e610e8 P a36b510d73684a5ea8a77d4745925790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78592a6111ce4606a8f33eed3d2ba83b in term 1.
I20260924 16:06:34.106624 11999 raft_consensus.cc:2454] T f5e1113468cd434d9a848d79f9e610e8 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78592a6111ce4606a8f33eed3d2ba83b in term 1.
I20260924 16:06:34.106887 11791 leader_election.cc:302] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78592a6111ce4606a8f33eed3d2ba83b, a36b510d73684a5ea8a77d4745925790; no voters:
I20260924 16:06:34.108167 12278 raft_consensus.cc:2781] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:34.108477 12278 raft_consensus.cc:686] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b [term 1 LEADER]: Becoming Leader. State: Replica: 78592a6111ce4606a8f33eed3d2ba83b, State: Running, Role: LEADER
I20260924 16:06:34.108851 12278 consensus_queue.cc:237] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:34.116086 11701 catalog_manager.cc:5495] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b reported cstate change: term changed from 0 to 1, leader changed from <none> to 78592a6111ce4606a8f33eed3d2ba83b (127.3.214.193). New cstate: current_term: 1 leader_uuid: "78592a6111ce4606a8f33eed3d2ba83b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } health_report { overall_health: UNKNOWN } } }
I20260924 16:06:34.144152 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:34.138989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44044 (local address 127.3.214.193:42201)
0924 16:06:34.139196 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:34.139202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:34.139265 (+ 63us) server_negotiation.cc:407] Connection header received
0924 16:06:34.139512 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:34.139516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:34.139609 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:34.139686 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:34.140066 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.140863 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.142223 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.142704 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.142819 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:34.143264 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:34.143399 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:34.143474 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:34.143696 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 16:06:34.143793 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 16:06:34.147380 12146 raft_consensus.cc:1261] T f5e1113468cd434d9a848d79f9e610e8 P a36b510d73684a5ea8a77d4745925790 [term 1 FOLLOWER]: Refusing update from remote peer 78592a6111ce4606a8f33eed3d2ba83b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:34.147648 11999 raft_consensus.cc:1261] T f5e1113468cd434d9a848d79f9e610e8 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 1 FOLLOWER]: Refusing update from remote peer 78592a6111ce4606a8f33eed3d2ba83b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:34.148253 12278 consensus_queue.cc:1035] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:34.148780 12278 consensus_queue.cc:1035] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b [LEADER]: Connected to new peer: Peer: permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:34.181066 11701 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:34.181406 11701 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:34.184870 11701 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a: Sending DeleteTablet for 3 replicas of tablet f5e1113468cd434d9a848d79f9e610e8
I20260924 16:06:34.185815 12120 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e1113468cd434d9a848d79f9e610e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:34 UTC) from {username='oryx', principal='oryx/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:55858
I20260924 16:06:34.186008 11980 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e1113468cd434d9a848d79f9e610e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:34 UTC) from {username='oryx', principal='oryx/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:60260
I20260924 16:06:34.186221 11838 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e1113468cd434d9a848d79f9e610e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:34 UTC) from {username='oryx', principal='oryx/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:44034
I20260924 16:06:34.186559 12281 tablet_replica.cc:330] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b: stopping tablet replica
I20260924 16:06:34.186563 12282 tablet_replica.cc:330] T f5e1113468cd434d9a848d79f9e610e8 P a36b510d73684a5ea8a77d4745925790: stopping tablet replica
I20260924 16:06:34.186779 12282 raft_consensus.cc:2229] T f5e1113468cd434d9a848d79f9e610e8 P a36b510d73684a5ea8a77d4745925790 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:06:34.186779 12281 raft_consensus.cc:2229] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:06:34.187043 12282 raft_consensus.cc:2258] T f5e1113468cd434d9a848d79f9e610e8 P a36b510d73684a5ea8a77d4745925790 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:34.187144 12281 raft_consensus.cc:2258] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:34.187350 12280 tablet_replica.cc:330] T f5e1113468cd434d9a848d79f9e610e8 P 575a2d7c7d554b3fbbe872b0a4601ca1: stopping tablet replica
I20260924 16:06:34.187541 12280 raft_consensus.cc:2229] T f5e1113468cd434d9a848d79f9e610e8 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:06:34.187755 12280 raft_consensus.cc:2258] T f5e1113468cd434d9a848d79f9e610e8 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:34.189079 12281 ts_tablet_manager.cc:1905] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:34.190599 11699 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:06:34.191411 11699 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:06:34.198853 12281 ts_tablet_manager.cc:1918] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:06:34.199134 12281 log.cc:1196] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-0/wal/wals/f5e1113468cd434d9a848d79f9e610e8
I20260924 16:06:34.199805 12281 ts_tablet_manager.cc:1939] T f5e1113468cd434d9a848d79f9e610e8 P 78592a6111ce4606a8f33eed3d2ba83b: Deleting consensus metadata
I20260924 16:06:34.200259 12120 tablet_service.cc:1467] Processing CreateTablet for tablet 401c7653d75e40029828d365ce356a47 (DEFAULT_TABLE table=test-table [id=5eb3012992624c48a50f37e99ee5218b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:34.200810 12120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 401c7653d75e40029828d365ce356a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:34.202590 11685 catalog_manager.cc:4841] TS 78592a6111ce4606a8f33eed3d2ba83b (127.3.214.193:42201): tablet f5e1113468cd434d9a848d79f9e610e8 (table test-table [id=b7f7e94df39146948a0a17275c2238a1]) successfully deleted
I20260924 16:06:34.203512 11980 tablet_service.cc:1467] Processing CreateTablet for tablet 401c7653d75e40029828d365ce356a47 (DEFAULT_TABLE table=test-table [id=5eb3012992624c48a50f37e99ee5218b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:34.203976 11980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 401c7653d75e40029828d365ce356a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:34.206993 11838 tablet_service.cc:1467] Processing CreateTablet for tablet 401c7653d75e40029828d365ce356a47 (DEFAULT_TABLE table=test-table [id=5eb3012992624c48a50f37e99ee5218b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:06:34.207479 11838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 401c7653d75e40029828d365ce356a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:06:34.211894 12217 tablet_bootstrap.cc:492] T 401c7653d75e40029828d365ce356a47 P 575a2d7c7d554b3fbbe872b0a4601ca1: Bootstrap starting.
I20260924 16:06:34.213683 12217 tablet_bootstrap.cc:654] T 401c7653d75e40029828d365ce356a47 P 575a2d7c7d554b3fbbe872b0a4601ca1: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:34.218204 12282 ts_tablet_manager.cc:1905] T f5e1113468cd434d9a848d79f9e610e8 P a36b510d73684a5ea8a77d4745925790: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:34.218874 12221 tablet_bootstrap.cc:492] T 401c7653d75e40029828d365ce356a47 P 78592a6111ce4606a8f33eed3d2ba83b: Bootstrap starting.
I20260924 16:06:34.220808 12221 tablet_bootstrap.cc:654] T 401c7653d75e40029828d365ce356a47 P 78592a6111ce4606a8f33eed3d2ba83b: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:34.226352 12282 ts_tablet_manager.cc:1918] T f5e1113468cd434d9a848d79f9e610e8 P a36b510d73684a5ea8a77d4745925790: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:06:34.228863 12280 ts_tablet_manager.cc:1905] T f5e1113468cd434d9a848d79f9e610e8 P 575a2d7c7d554b3fbbe872b0a4601ca1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:34.235960 12221 tablet_bootstrap.cc:492] T 401c7653d75e40029828d365ce356a47 P 78592a6111ce4606a8f33eed3d2ba83b: No bootstrap required, opened a new log
I20260924 16:06:34.236208 12221 ts_tablet_manager.cc:1397] T 401c7653d75e40029828d365ce356a47 P 78592a6111ce4606a8f33eed3d2ba83b: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.004s
I20260924 16:06:34.236487 12280 ts_tablet_manager.cc:1918] T f5e1113468cd434d9a848d79f9e610e8 P 575a2d7c7d554b3fbbe872b0a4601ca1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:06:34.236706 12280 log.cc:1196] T f5e1113468cd434d9a848d79f9e610e8 P 575a2d7c7d554b3fbbe872b0a4601ca1: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/f5e1113468cd434d9a848d79f9e610e8
I20260924 16:06:34.237244 12280 ts_tablet_manager.cc:1939] T f5e1113468cd434d9a848d79f9e610e8 P 575a2d7c7d554b3fbbe872b0a4601ca1: Deleting consensus metadata
I20260924 16:06:34.238937 12221 raft_consensus.cc:348] T 401c7653d75e40029828d365ce356a47 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:34.238462 12217 tablet_bootstrap.cc:492] T 401c7653d75e40029828d365ce356a47 P 575a2d7c7d554b3fbbe872b0a4601ca1: No bootstrap required, opened a new log
I20260924 16:06:34.239261 12221 raft_consensus.cc:374] T 401c7653d75e40029828d365ce356a47 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:34.239362 12221 raft_consensus.cc:729] T 401c7653d75e40029828d365ce356a47 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78592a6111ce4606a8f33eed3d2ba83b, State: Initialized, Role: FOLLOWER
I20260924 16:06:34.239533 12217 ts_tablet_manager.cc:1397] T 401c7653d75e40029828d365ce356a47 P 575a2d7c7d554b3fbbe872b0a4601ca1: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.000s
I20260924 16:06:34.239670 12221 consensus_queue.cc:260] T 401c7653d75e40029828d365ce356a47 P 78592a6111ce4606a8f33eed3d2ba83b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:34.240454 12217 raft_consensus.cc:348] T 401c7653d75e40029828d365ce356a47 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:34.240758 12217 raft_consensus.cc:374] T 401c7653d75e40029828d365ce356a47 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:34.240911 12217 raft_consensus.cc:729] T 401c7653d75e40029828d365ce356a47 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 575a2d7c7d554b3fbbe872b0a4601ca1, State: Initialized, Role: FOLLOWER
I20260924 16:06:34.241247 12217 consensus_queue.cc:260] T 401c7653d75e40029828d365ce356a47 P 575a2d7c7d554b3fbbe872b0a4601ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:34.241878 12217 ts_tablet_manager.cc:1428] T 401c7653d75e40029828d365ce356a47 P 575a2d7c7d554b3fbbe872b0a4601ca1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 16:06:34.226500 12218 tablet_bootstrap.cc:492] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790: Bootstrap starting.
I20260924 16:06:34.242322 12282 log.cc:1196] T f5e1113468cd434d9a848d79f9e610e8 P a36b510d73684a5ea8a77d4745925790: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/f5e1113468cd434d9a848d79f9e610e8
I20260924 16:06:34.243026 12282 ts_tablet_manager.cc:1939] T f5e1113468cd434d9a848d79f9e610e8 P a36b510d73684a5ea8a77d4745925790: Deleting consensus metadata
I20260924 16:06:34.244279 12218 tablet_bootstrap.cc:654] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:34.244912 11688 catalog_manager.cc:4841] TS a36b510d73684a5ea8a77d4745925790 (127.3.214.195:37245): tablet f5e1113468cd434d9a848d79f9e610e8 (table test-table [id=b7f7e94df39146948a0a17275c2238a1]) successfully deleted
I20260924 16:06:34.245828 11685 catalog_manager.cc:4841] TS 575a2d7c7d554b3fbbe872b0a4601ca1 (127.3.214.194:43513): tablet f5e1113468cd434d9a848d79f9e610e8 (table test-table [id=b7f7e94df39146948a0a17275c2238a1]) successfully deleted
I20260924 16:06:34.250342 12218 tablet_bootstrap.cc:492] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790: No bootstrap required, opened a new log
I20260924 16:06:34.250543 12218 ts_tablet_manager.cc:1397] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20260924 16:06:34.251252 12218 raft_consensus.cc:348] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:34.251503 12218 raft_consensus.cc:374] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:34.251583 12218 raft_consensus.cc:729] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a36b510d73684a5ea8a77d4745925790, State: Initialized, Role: FOLLOWER
I20260924 16:06:34.251843 12218 consensus_queue.cc:260] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:34.253664 12218 ts_tablet_manager.cc:1428] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 16:06:34.256191 12221 ts_tablet_manager.cc:1428] T 401c7653d75e40029828d365ce356a47 P 78592a6111ce4606a8f33eed3d2ba83b: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20260924 16:06:34.340086 12224 raft_consensus.cc:482] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:06:34.340390 12224 raft_consensus.cc:504] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:34.341288 12224 leader_election.cc:288] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78592a6111ce4606a8f33eed3d2ba83b (127.3.214.193:42201), 575a2d7c7d554b3fbbe872b0a4601ca1 (127.3.214.194:43513)
I20260924 16:06:34.342232 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "401c7653d75e40029828d365ce356a47" candidate_uuid: "a36b510d73684a5ea8a77d4745925790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78592a6111ce4606a8f33eed3d2ba83b" is_pre_election: true
I20260924 16:06:34.342381 12000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "401c7653d75e40029828d365ce356a47" candidate_uuid: "a36b510d73684a5ea8a77d4745925790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" is_pre_election: true
I20260924 16:06:34.342547 11863 raft_consensus.cc:2454] T 401c7653d75e40029828d365ce356a47 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a36b510d73684a5ea8a77d4745925790 in term 0.
I20260924 16:06:34.342746 12000 raft_consensus.cc:2454] T 401c7653d75e40029828d365ce356a47 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a36b510d73684a5ea8a77d4745925790 in term 0.
I20260924 16:06:34.343469 12070 leader_election.cc:302] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 575a2d7c7d554b3fbbe872b0a4601ca1, a36b510d73684a5ea8a77d4745925790; no voters:
I20260924 16:06:34.344142 12224 raft_consensus.cc:2781] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:06:34.344278 12224 raft_consensus.cc:482] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:06:34.344404 12224 raft_consensus.cc:3037] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:34.348183 12224 raft_consensus.cc:504] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:34.349066 12224 leader_election.cc:288] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790 [CANDIDATE]: Term 1 election: Requested vote from peers 78592a6111ce4606a8f33eed3d2ba83b (127.3.214.193:42201), 575a2d7c7d554b3fbbe872b0a4601ca1 (127.3.214.194:43513)
I20260924 16:06:34.349922 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "401c7653d75e40029828d365ce356a47" candidate_uuid: "a36b510d73684a5ea8a77d4745925790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78592a6111ce4606a8f33eed3d2ba83b"
I20260924 16:06:34.350008 12000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "401c7653d75e40029828d365ce356a47" candidate_uuid: "a36b510d73684a5ea8a77d4745925790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1"
I20260924 16:06:34.350268 11863 raft_consensus.cc:3037] T 401c7653d75e40029828d365ce356a47 P 78592a6111ce4606a8f33eed3d2ba83b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:34.350366 12000 raft_consensus.cc:3037] T 401c7653d75e40029828d365ce356a47 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:34.353796 11863 raft_consensus.cc:2454] T 401c7653d75e40029828d365ce356a47 P 78592a6111ce4606a8f33eed3d2ba83b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36b510d73684a5ea8a77d4745925790 in term 1.
I20260924 16:06:34.353941 12000 raft_consensus.cc:2454] T 401c7653d75e40029828d365ce356a47 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36b510d73684a5ea8a77d4745925790 in term 1.
I20260924 16:06:34.354657 12070 leader_election.cc:302] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78592a6111ce4606a8f33eed3d2ba83b, a36b510d73684a5ea8a77d4745925790; no voters:
I20260924 16:06:34.355811 12224 raft_consensus.cc:2781] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:34.356447 12224 raft_consensus.cc:686] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790 [term 1 LEADER]: Becoming Leader. State: Replica: a36b510d73684a5ea8a77d4745925790, State: Running, Role: LEADER
I20260924 16:06:34.356974 12224 consensus_queue.cc:237] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } }
I20260924 16:06:34.361768 11699 catalog_manager.cc:5495] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790 reported cstate change: term changed from 0 to 1, leader changed from <none> to a36b510d73684a5ea8a77d4745925790 (127.3.214.195). New cstate: current_term: 1 leader_uuid: "a36b510d73684a5ea8a77d4745925790" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a36b510d73684a5ea8a77d4745925790" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37245 } health_report { overall_health: HEALTHY } } }
I20260924 16:06:34.413479 12306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:34.406523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55884 (local address 127.3.214.195:37245)
0924 16:06:34.407373 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:34.407378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:34.407394 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:06:34.407474 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:34.407479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:34.407569 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:34.407663 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:34.408280 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.409130 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.411216 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.411734 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.411848 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:34.412467 (+ 619us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:34.412602 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:34.412711 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:34.412957 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 16:06:34.413061 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":602,"threads_started":1}
I20260924 16:06:34.432940 12306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:34.425203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:58989 (local address 127.3.214.195:37245)
0924 16:06:34.426330 (+ 1127us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:34.426338 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:34.426373 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:06:34.426455 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:34.426460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:34.426565 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:34.426696 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:06:34.427344 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.428457 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.430543 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.431286 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.431430 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:34.432015 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:06:34.432174 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:06:34.432267 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:34.432464 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 16:06:34.432565 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":962}
I20260924 16:06:34.433626 12309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:34.425211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:37245 (local address 127.3.214.195:58989)
0924 16:06:34.425951 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:34.425974 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:34.426164 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:34.426725 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:34.426731 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:34.426746 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:06:34.427122 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:34.427135 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.428643 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.428650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:34.430314 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:34.430333 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.431536 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.431544 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:34.431743 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:34.431809 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:06:34.432305 (+ 496us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:06:34.432318 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 16:06:34.432437 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 16:06:34.432727 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":389,"threads_started":1}
I20260924 16:06:34.451035 11863 raft_consensus.cc:1261] T 401c7653d75e40029828d365ce356a47 P 78592a6111ce4606a8f33eed3d2ba83b [term 1 FOLLOWER]: Refusing update from remote peer a36b510d73684a5ea8a77d4745925790: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:34.451907 12000 raft_consensus.cc:1261] T 401c7653d75e40029828d365ce356a47 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 1 FOLLOWER]: Refusing update from remote peer a36b510d73684a5ea8a77d4745925790: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:06:34.451824 12310 consensus_queue.cc:1035] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78592a6111ce4606a8f33eed3d2ba83b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 42201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:34.452485 12236 consensus_queue.cc:1035] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790 [LEADER]: Connected to new peer: Peer: permanent_uuid: "575a2d7c7d554b3fbbe872b0a4601ca1" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:06:34.531138 11699 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:34.531484 11699 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:06:34.535503 11699 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 623e994a63f541cbb502fa94e1be836a: Sending DeleteTablet for 3 replicas of tablet 401c7653d75e40029828d365ce356a47
I20260924 16:06:34.536767 11980 tablet_service.cc:1514] Processing DeleteTablet for tablet 401c7653d75e40029828d365ce356a47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:34 UTC) from {username='oryx', principal='oryx/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:60260
I20260924 16:06:34.537132 12280 tablet_replica.cc:330] T 401c7653d75e40029828d365ce356a47 P 575a2d7c7d554b3fbbe872b0a4601ca1: stopping tablet replica
I20260924 16:06:34.537192 11838 tablet_service.cc:1514] Processing DeleteTablet for tablet 401c7653d75e40029828d365ce356a47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:34 UTC) from {username='oryx', principal='oryx/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:44034
I20260924 16:06:34.537334 12280 raft_consensus.cc:2229] T 401c7653d75e40029828d365ce356a47 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:06:34.537577 12280 raft_consensus.cc:2258] T 401c7653d75e40029828d365ce356a47 P 575a2d7c7d554b3fbbe872b0a4601ca1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:34.537658 12281 tablet_replica.cc:330] T 401c7653d75e40029828d365ce356a47 P 78592a6111ce4606a8f33eed3d2ba83b: stopping tablet replica
I20260924 16:06:34.537868 12281 raft_consensus.cc:2229] T 401c7653d75e40029828d365ce356a47 P 78592a6111ce4606a8f33eed3d2ba83b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:06:34.537927 12120 tablet_service.cc:1514] Processing DeleteTablet for tablet 401c7653d75e40029828d365ce356a47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:06:34 UTC) from {username='oryx', principal='oryx/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:55858
I20260924 16:06:34.538146 12281 raft_consensus.cc:2258] T 401c7653d75e40029828d365ce356a47 P 78592a6111ce4606a8f33eed3d2ba83b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:34.538278 12282 tablet_replica.cc:330] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790: stopping tablet replica
I20260924 16:06:34.538465 12282 raft_consensus.cc:2229] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:06:34.538842 12282 raft_consensus.cc:2258] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:06:34.539522 12281 ts_tablet_manager.cc:1905] T 401c7653d75e40029828d365ce356a47 P 78592a6111ce4606a8f33eed3d2ba83b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:34.539908 12280 ts_tablet_manager.cc:1905] T 401c7653d75e40029828d365ce356a47 P 575a2d7c7d554b3fbbe872b0a4601ca1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:34.540602 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 11772
I20260924 16:06:34.543119 12282 ts_tablet_manager.cc:1905] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:06:34.548612 12281 ts_tablet_manager.cc:1918] T 401c7653d75e40029828d365ce356a47 P 78592a6111ce4606a8f33eed3d2ba83b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:06:34.548840 12281 log.cc:1196] T 401c7653d75e40029828d365ce356a47 P 78592a6111ce4606a8f33eed3d2ba83b: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-0/wal/wals/401c7653d75e40029828d365ce356a47
I20260924 16:06:34.549360 12281 ts_tablet_manager.cc:1939] T 401c7653d75e40029828d365ce356a47 P 78592a6111ce4606a8f33eed3d2ba83b: Deleting consensus metadata
I20260924 16:06:34.551479 11685 catalog_manager.cc:4841] TS 78592a6111ce4606a8f33eed3d2ba83b (127.3.214.193:42201): tablet 401c7653d75e40029828d365ce356a47 (table test-table [id=5eb3012992624c48a50f37e99ee5218b]) successfully deleted
I20260924 16:06:34.552613 12282 ts_tablet_manager.cc:1918] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:06:34.552812 12282 log.cc:1196] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/401c7653d75e40029828d365ce356a47
I20260924 16:06:34.553303 12282 ts_tablet_manager.cc:1939] T 401c7653d75e40029828d365ce356a47 P a36b510d73684a5ea8a77d4745925790: Deleting consensus metadata
I20260924 16:06:34.554840 11688 catalog_manager.cc:4841] TS a36b510d73684a5ea8a77d4745925790 (127.3.214.195:37245): tablet 401c7653d75e40029828d365ce356a47 (table test-table [id=5eb3012992624c48a50f37e99ee5218b]) successfully deleted
I20260924 16:06:34.555269 12280 ts_tablet_manager.cc:1918] T 401c7653d75e40029828d365ce356a47 P 575a2d7c7d554b3fbbe872b0a4601ca1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:06:34.555509 12280 log.cc:1196] T 401c7653d75e40029828d365ce356a47 P 575a2d7c7d554b3fbbe872b0a4601ca1: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/401c7653d75e40029828d365ce356a47
I20260924 16:06:34.555932 12280 ts_tablet_manager.cc:1939] T 401c7653d75e40029828d365ce356a47 P 575a2d7c7d554b3fbbe872b0a4601ca1: Deleting consensus metadata
I20260924 16:06:34.557691 11685 catalog_manager.cc:4841] TS 575a2d7c7d554b3fbbe872b0a4601ca1 (127.3.214.194:43513): tablet 401c7653d75e40029828d365ce356a47 (table test-table [id=5eb3012992624c48a50f37e99ee5218b]) successfully deleted
Sep 24 16:06:34 dist-test-slave-1gc3 krb5kdc[11655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.214.193@KRBTEST.COM
I20260924 16:06:34.570935 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:34.542455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.3.214.193:42201)
0924 16:06:34.542671 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:34.542679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:34.552203 (+ 9524us) server_negotiation.cc:407] Connection header received
0924 16:06:34.552431 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:34.552437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:34.552527 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:34.552632 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:34.554850 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.555912 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.557284 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.557857 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.557973 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:34.563509 (+ 5536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:34.563537 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:34.563563 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:34.563622 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:34.567145 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:34.568267 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:34.568277 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:34.568279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:34.568631 (+ 352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:34.569139 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:34.569143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:34.569146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:34.569601 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:34.569844 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:34.570376 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 16:06:34.570646 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:06:34.571887 11904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:34.709124 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 11772
I20260924 16:06:34.737560 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 11913
Sep 24 16:06:34 dist-test-slave-1gc3 krb5kdc[11655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.214.194@KRBTEST.COM
I20260924 16:06:34.755812 12317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:34.738737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36194 (local address 127.3.214.194:43513)
0924 16:06:34.739312 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:34.739318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:34.739337 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:06:34.739401 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:34.739405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:34.739479 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:34.739582 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:34.741217 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.742178 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.743699 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.744181 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.744305 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:34.748969 (+ 4664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:34.748990 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:34.749012 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:34.749057 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:34.752472 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:34.753378 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:34.753387 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:34.753390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:34.753566 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:34.754042 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:34.754046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:34.754048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:34.754518 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:34.754824 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:34.755201 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 16:06:34.755463 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":301,"threads_started":1}
I20260924 16:06:34.756479 12045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:34.869788 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 11913
I20260924 16:06:34.892840 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 12054
Sep 24 16:06:34 dist-test-slave-1gc3 krb5kdc[11655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.214.195@KRBTEST.COM
I20260924 16:06:34.910568 12306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:34.893945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43488 (local address 127.3.214.195:37245)
0924 16:06:34.894177 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:34.894184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:34.894207 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:06:34.894284 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:34.894288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:34.894368 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:34.894477 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:34.895890 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.896774 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.898248 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:34.898721 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:34.898827 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:34.903624 (+ 4797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:34.903653 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:34.903682 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:34.903734 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:34.907109 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:34.908045 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:34.908055 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:34.908057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:34.908248 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:34.908890 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:34.908893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:34.908896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:34.909363 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:34.909670 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:34.909908 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 16:06:34.910179 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:06:34.911367 12187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:35.075395 12320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:35.073775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:43513 (local address 127.3.214.195:51057)
0924 16:06:35.074990 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:35.075176 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.194:43513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1013,"thread_start_us":833,"threads_started":1}
I20260924 16:06:35.075953 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:35.074981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:42201 (local address 127.3.214.195:45063)
0924 16:06:35.075501 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:35.075657 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:42201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":271,"threads_started":1}
W20260924 16:06:35.075930 12070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.214.194:43513: connect: Connection refused (error 111)
I20260924 16:06:35.076700 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 12054
I20260924 16:06:35.108395 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 11668
I20260924 16:06:35.110018 11741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:35.252725 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 11668
2026-09-24T16:06:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7520 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:35 dist-test-slave-1gc3 krb5kdc[12326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:35 dist-test-slave-1gc3 krb5kdc[12326](info): set up 2 sockets
Sep 24 16:06:35 dist-test-slave-1gc3 krb5kdc[12326](info): commencing operation
krb5kdc: starting...
W20260924 16:06:36.312511 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:06:36 dist-test-slave-1gc3 krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:06:36.590916 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:40031
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:40031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:36.976384 12340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:36.976722 12340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:36.976830 12340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:36.984869 12340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:36.985008 12340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:36.985057 12340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:36.985164 12340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:36.985242 12340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:36.996016 12340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:40031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:40031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:36.997489 12340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:36.999974 12340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:37.014482 12345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:37.014570 12348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:37.014511 12346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:37.015803 12340 server_base.cc:1034] running on GCE node
I20260924 16:06:37.016745 12340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:37.018559 12340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:37.019886 12340 hybrid_clock.cc:648] HybridClock initialized: now 1790265997019797 us; error 64 us; skew 500 ppm
I20260924 16:06:37.020350 12340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T16:06:37Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1839 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:06:37 dist-test-slave-1gc3 krb5kdc[12354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:06:37 dist-test-slave-1gc3 krb5kdc[12354](info): set up 2 sockets
Sep 24 16:06:37 dist-test-slave-1gc3 krb5kdc[12354](info): commencing operation
krb5kdc: starting...
W20260924 16:06:38.170557 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.006s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:06:38 dist-test-slave-1gc3 krb5kdc[12354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:06:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:06:38.531689 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:44483
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:44483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:38.944059 12367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:38.944471 12367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:38.944583 12367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:38.953575 12367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:38.953764 12367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:38.953820 12367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:38.953887 12367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:38.953936 12367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:38.965955 12367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:44483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:44483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:38.967880 12367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:38.969578 12367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:38.984503 12372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:38.984787 12375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:38.984596 12373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:38.985621 12367 server_base.cc:1034] running on GCE node
I20260924 16:06:38.986734 12367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:38.988503 12367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:38.989766 12367 hybrid_clock.cc:648] HybridClock initialized: now 1790265998989680 us; error 65 us; skew 500 ppm
I20260924 16:06:38.990248 12367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:38 dist-test-slave-1gc3 krb5kdc[12354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265998, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:38.997310 12367 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:06:38.999475 12367 webserver.cc:460] Webserver started at http://127.3.214.254:45433/ using document root <none> and password file <none>
I20260924 16:06:39.000396 12367 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:39.000530 12367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:39.000883 12367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:39.004766 12367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "065e760ec0924ea28d50eef9c1ae9cdb"
format_stamp: "Formatted at 2026-09-24 16:06:38 on dist-test-slave-1gc3"
server_key: "1af1c683aa8a72f4fda097193b5f9af9"
server_key_iv: "a1063925c7703f7a0ada898b2dde1284"
server_key_version: "encryptionkey@0"
I20260924 16:06:39.005754 12367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "065e760ec0924ea28d50eef9c1ae9cdb"
format_stamp: "Formatted at 2026-09-24 16:06:38 on dist-test-slave-1gc3"
server_key: "1af1c683aa8a72f4fda097193b5f9af9"
server_key_iv: "a1063925c7703f7a0ada898b2dde1284"
server_key_version: "encryptionkey@0"
I20260924 16:06:39.012705 12367 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 16:06:39.017867 12382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:39.020206 12367 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260924 16:06:39.020484 12367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "065e760ec0924ea28d50eef9c1ae9cdb"
format_stamp: "Formatted at 2026-09-24 16:06:38 on dist-test-slave-1gc3"
server_key: "1af1c683aa8a72f4fda097193b5f9af9"
server_key_iv: "a1063925c7703f7a0ada898b2dde1284"
server_key_version: "encryptionkey@0"
I20260924 16:06:39.020762 12367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:39.037454 12367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:39.043321 12367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:39.043635 12367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:39.068765 12433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:44483 every 8 connection(s)
I20260924 16:06:39.068758 12367 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:44483
I20260924 16:06:39.070779 12367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:39.071482 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 12367
I20260924 16:06:39.071717 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:06:39.072533 3931 external_mini_cluster.cc:1428] Setting key 30dbeca980a058ded78abd331175b0d3
I20260924 16:06:39.076913 12434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 16:06:39 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:06:39.092968 12434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb: Bootstrap starting.
I20260924 16:06:39.098551 12434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb: Neither blocks nor log segments found. Creating new log.
I20260924 16:06:39.100303 12434 log.cc:824] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:39.103271 12437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:39.075058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55630 (local address 127.3.214.254:44483)
0924 16:06:39.076106 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:39.076128 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:39.076196 (+ 68us) server_negotiation.cc:407] Connection header received
0924 16:06:39.078868 (+ 2672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:39.078907 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:39.079534 (+ 627us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:39.080094 (+ 560us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:39.081741 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:39.083459 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:39.085498 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:39.086531 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:39.086742 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:39.092433 (+ 5691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:39.092448 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:39.092481 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:39.092554 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:39.097102 (+ 4548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:39.098247 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:39.098258 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:39.098269 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:39.098577 (+ 308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:39.099064 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:39.099069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:39.099071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:39.099915 (+ 844us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:39.100315 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:39.100921 (+ 606us) server_negotiation.cc:299] Negotiation successful
0924 16:06:39.101592 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":418,"threads_started":1}
I20260924 16:06:39.104764 12434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb: No bootstrap required, opened a new log
I20260924 16:06:39.113694 12434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065e760ec0924ea28d50eef9c1ae9cdb" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 44483 } }
I20260924 16:06:39.114143 12434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:06:39.114238 12434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 065e760ec0924ea28d50eef9c1ae9cdb, State: Initialized, Role: FOLLOWER
I20260924 16:06:39.114856 12434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065e760ec0924ea28d50eef9c1ae9cdb" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 44483 } }
I20260924 16:06:39.115101 12434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:39.115239 12434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:39.115419 12434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:06:39.118171 12434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065e760ec0924ea28d50eef9c1ae9cdb" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 44483 } }
I20260924 16:06:39.118755 12434 leader_election.cc:302] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 065e760ec0924ea28d50eef9c1ae9cdb; no voters:
I20260924 16:06:39.119423 12434 leader_election.cc:288] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:06:39.119663 12439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:06:39.120545 12439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [term 1 LEADER]: Becoming Leader. State: Replica: 065e760ec0924ea28d50eef9c1ae9cdb, State: Running, Role: LEADER
I20260924 16:06:39.121258 12439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065e760ec0924ea28d50eef9c1ae9cdb" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 44483 } }
I20260924 16:06:39.121764 12434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:39.125976 12441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 065e760ec0924ea28d50eef9c1ae9cdb. Latest consensus state: current_term: 1 leader_uuid: "065e760ec0924ea28d50eef9c1ae9cdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065e760ec0924ea28d50eef9c1ae9cdb" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 44483 } } }
I20260924 16:06:39.126535 12441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:39.125870 12440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "065e760ec0924ea28d50eef9c1ae9cdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065e760ec0924ea28d50eef9c1ae9cdb" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 44483 } } }
I20260924 16:06:39.127250 12440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:39.127923 12448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:39.135582 12448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:39.149403 12448 catalog_manager.cc:1347] Generated new cluster ID: 2d421ce8e260487081926e11db19377f
I20260924 16:06:39.149554 12448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:39.163643 12448 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:06:39.165105 12448 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:39.186843 12448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb: Generated new TSK 0
I20260924 16:06:39.187861 12448 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.214.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
I20260924 16:06:39.262910 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:44483
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:39.651625 12460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:39.652035 12460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:39.652185 12460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:39.660701 12460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:39.660869 12460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:39.661038 12460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:06:39.672482 12460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:44483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:39.674324 12460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:39.676004 12460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:39.692006 12466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:39.692095 12465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:39.693120 12468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:39.695185 12460 server_base.cc:1034] running on GCE node
I20260924 16:06:39.696048 12460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:39.697299 12460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:39.698549 12460 hybrid_clock.cc:648] HybridClock initialized: now 1790265999698487 us; error 37 us; skew 500 ppm
I20260924 16:06:39.698989 12460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:39 dist-test-slave-1gc3 krb5kdc[12354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265999, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:39.703158 12460 init.cc:377] Logged in from keytab as kudu/127.3.214.193@KRBTEST.COM (short username kudu)
I20260924 16:06:39.705225 12460 webserver.cc:460] Webserver started at http://127.3.214.193:42331/ using document root <none> and password file <none>
I20260924 16:06:39.706226 12460 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:39.706388 12460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:39.706771 12460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:39.710680 12460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "3e4db2ce4d4b417ebd7419571660195f"
format_stamp: "Formatted at 2026-09-24 16:06:39 on dist-test-slave-1gc3"
server_key: "465f4394810055b81cc635ca99b22871"
server_key_iv: "fcb1ae3e7418e1b78f40c0e6ffd899ac"
server_key_version: "encryptionkey@0"
I20260924 16:06:39.711714 12460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "3e4db2ce4d4b417ebd7419571660195f"
format_stamp: "Formatted at 2026-09-24 16:06:39 on dist-test-slave-1gc3"
server_key: "465f4394810055b81cc635ca99b22871"
server_key_iv: "fcb1ae3e7418e1b78f40c0e6ffd899ac"
server_key_version: "encryptionkey@0"
I20260924 16:06:39.718984 12460 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260924 16:06:39.724308 12475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:39.726526 12460 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:06:39.726784 12460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "3e4db2ce4d4b417ebd7419571660195f"
format_stamp: "Formatted at 2026-09-24 16:06:39 on dist-test-slave-1gc3"
server_key: "465f4394810055b81cc635ca99b22871"
server_key_iv: "fcb1ae3e7418e1b78f40c0e6ffd899ac"
server_key_version: "encryptionkey@0"
I20260924 16:06:39.727062 12460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:39.778537 12460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:39.784375 12460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:39.784746 12460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:39.786077 12460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:39.788021 12460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:39.788116 12460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:39.788250 12460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:39.788314 12460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:39.827725 12460 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:34195
I20260924 16:06:39.827766 12588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:34195 every 8 connection(s)
I20260924 16:06:39.829671 12460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:06:39.831247 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 12460
I20260924 16:06:39.831501 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:06:39.832422 3931 external_mini_cluster.cc:1428] Setting key 6c7569beab2a7f9236ec1fe0b398025b
Sep 24 16:06:39 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265999, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:06:39.857570 12591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:39.833579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:60695 (local address 127.3.214.254:44483)
0924 16:06:39.834275 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:39.834282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:39.835543 (+ 1261us) server_negotiation.cc:407] Connection header received
0924 16:06:39.837802 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:39.837808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:39.837919 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:39.838076 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:39.840107 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:39.841206 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:39.843699 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:39.844340 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:39.844524 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:39.849731 (+ 5207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:39.849739 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:39.849743 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:39.849797 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:39.852851 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:39.854387 (+ 1536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:39.854393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:39.854395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:39.854587 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:39.855241 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:39.855246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:39.855248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:39.855648 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:39.855791 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:39.856998 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0924 16:06:39.857245 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":360,"threads_started":1}
I20260924 16:06:39.859411 12592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:39.834315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.193:60695)
0924 16:06:39.835266 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:39.835333 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:39.837411 (+ 2078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:39.838471 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:39.838481 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:39.839117 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:39.839845 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:39.839875 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:39.841520 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:39.841526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:39.843463 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:39.843487 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:39.844654 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:39.844661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:39.845170 (+ 509us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:39.845961 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:39.845989 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:39.849091 (+ 3102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:39.853134 (+ 4043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:39.853151 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:39.853181 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:39.854083 (+ 902us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:39.854728 (+ 645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:39.854732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:39.854736 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:39.855064 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:39.855799 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:39.855809 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:39.856419 (+ 610us) client_negotiation.cc:769] Sending connection context
0924 16:06:39.856776 (+ 357us) client_negotiation.cc:240] Negotiation successful
0924 16:06:39.857502 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":368,"threads_started":1}
I20260924 16:06:39.862046 12589 heartbeater.cc:347] Connected to a master server at 127.3.214.254:44483
I20260924 16:06:39.862586 12589 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:39.863747 12589 heartbeater.cc:511] Master 127.3.214.254:44483 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.214.194@KRBTEST.COM; defaulting to no policy
I20260924 16:06:39.866906 12399 ts_manager.cc:194] Registered new tserver with Master: 3e4db2ce4d4b417ebd7419571660195f (127.3.214.193:34195)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.194@KRBTEST.COM" created.
I20260924 16:06:39.869099 12399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:60695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
I20260924 16:06:39.902624 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:44483
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:40.283809 12595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:40.284161 12595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:40.284294 12595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:40.292280 12595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:40.292440 12595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:40.292599 12595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:06:40.304625 12595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:44483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:40.306177 12595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:40.307961 12595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:40.323560 12603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:40.323522 12600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:40.324364 12595 server_base.cc:1034] running on GCE node
W20260924 16:06:40.324747 12601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:40.325592 12595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:40.326876 12595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:40.328161 12595 hybrid_clock.cc:648] HybridClock initialized: now 1790266000328083 us; error 65 us; skew 500 ppm
I20260924 16:06:40.328565 12595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:40 dist-test-slave-1gc3 krb5kdc[12354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266000, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:40.332710 12595 init.cc:377] Logged in from keytab as kudu/127.3.214.194@KRBTEST.COM (short username kudu)
I20260924 16:06:40.334846 12595 webserver.cc:460] Webserver started at http://127.3.214.194:35235/ using document root <none> and password file <none>
I20260924 16:06:40.335827 12595 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:40.335968 12595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:40.336338 12595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:40.340216 12595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "f731f5d5ee3c4f2591ab8b6c3fb245ed"
format_stamp: "Formatted at 2026-09-24 16:06:40 on dist-test-slave-1gc3"
server_key: "7f7db02cf8568469fbfa197a75ab7b5f"
server_key_iv: "44a7e9601e011652e3ae8c52c839e5a9"
server_key_version: "encryptionkey@0"
I20260924 16:06:40.341255 12595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "f731f5d5ee3c4f2591ab8b6c3fb245ed"
format_stamp: "Formatted at 2026-09-24 16:06:40 on dist-test-slave-1gc3"
server_key: "7f7db02cf8568469fbfa197a75ab7b5f"
server_key_iv: "44a7e9601e011652e3ae8c52c839e5a9"
server_key_version: "encryptionkey@0"
I20260924 16:06:40.348026 12595 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260924 16:06:40.352973 12610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:40.354966 12595 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:06:40.355181 12595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "f731f5d5ee3c4f2591ab8b6c3fb245ed"
format_stamp: "Formatted at 2026-09-24 16:06:40 on dist-test-slave-1gc3"
server_key: "7f7db02cf8568469fbfa197a75ab7b5f"
server_key_iv: "44a7e9601e011652e3ae8c52c839e5a9"
server_key_version: "encryptionkey@0"
I20260924 16:06:40.355434 12595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:40.376804 12595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:40.382462 12595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:40.382823 12595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:40.384064 12595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:40.385782 12595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:40.385869 12595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:40.385990 12595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:40.386047 12595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:40.425613 12595 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:42951
I20260924 16:06:40.425699 12723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:42951 every 8 connection(s)
I20260924 16:06:40.427348 12595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:06:40.433218 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 12595
I20260924 16:06:40.433447 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:06:40.434059 3931 external_mini_cluster.cc:1428] Setting key 55579a06d27cae43d1d033505f815175
Sep 24 16:06:40 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266000, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:06:40.454330 12726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:40.430537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:54951 (local address 127.3.214.254:44483)
0924 16:06:40.431082 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:40.431087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:40.432462 (+ 1375us) server_negotiation.cc:407] Connection header received
0924 16:06:40.435273 (+ 2811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:40.435279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:40.435395 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:40.435500 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:40.437522 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:40.438454 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:40.440754 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:40.441308 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:40.441518 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:40.446536 (+ 5018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:40.446544 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:40.446548 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:40.446600 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:40.450080 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:40.451265 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:40.451271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:40.451274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:40.451467 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:40.452078 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:40.452083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:40.452085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:40.452500 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:40.452661 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:40.453632 (+ 971us) server_negotiation.cc:299] Negotiation successful
0924 16:06:40.453915 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":275,"threads_started":1}
I20260924 16:06:40.455593 12727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:40.431210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.194:54951)
0924 16:06:40.432245 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:40.432310 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:40.434908 (+ 2598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:40.435809 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:40.435819 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:40.436418 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:40.437279 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:40.437307 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:40.438650 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:40.438657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:40.440535 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:40.440555 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:40.441831 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:40.441842 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:40.442228 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:40.443115 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:40.443154 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:40.446052 (+ 2898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:40.450360 (+ 4308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:40.450372 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:40.450392 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:40.451094 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:40.451688 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:40.451693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:40.451696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:40.451949 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:40.452687 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:40.452695 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:40.453197 (+ 502us) client_negotiation.cc:769] Sending connection context
0924 16:06:40.453508 (+ 311us) client_negotiation.cc:240] Negotiation successful
0924 16:06:40.454064 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":417,"threads_started":1}
I20260924 16:06:40.457525 12724 heartbeater.cc:347] Connected to a master server at 127.3.214.254:44483
I20260924 16:06:40.457922 12724 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:40.458873 12724 heartbeater.cc:511] Master 127.3.214.254:44483 requested a full tablet report, sending...
I20260924 16:06:40.460937 12399 ts_manager.cc:194] Registered new tserver with Master: f731f5d5ee3c4f2591ab8b6c3fb245ed (127.3.214.194:42951)
WARNING: no policy specified for kudu/127.3.214.195@KRBTEST.COM; defaulting to no policy
I20260924 16:06:40.461951 12399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:54951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
I20260924 16:06:40.494598 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:44483
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:06:40.873384 12589 heartbeater.cc:503] Master 127.3.214.254:44483 was elected leader, sending a full tablet report...
W20260924 16:06:40.897092 12730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:40.897508 12730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:40.897663 12730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:40.906075 12730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:40.906307 12730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:40.906488 12730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:06:40.918128 12730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:44483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:40.920094 12730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:40.921906 12730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:40.939079 12738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:40.939141 12736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:40.939153 12735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:40.940099 12730 server_base.cc:1034] running on GCE node
I20260924 16:06:40.940974 12730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:40.942353 12730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:40.943593 12730 hybrid_clock.cc:648] HybridClock initialized: now 1790266000943520 us; error 76 us; skew 500 ppm
I20260924 16:06:40.944000 12730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:40 dist-test-slave-1gc3 krb5kdc[12354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266000, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:40.948290 12730 init.cc:377] Logged in from keytab as kudu/127.3.214.195@KRBTEST.COM (short username kudu)
I20260924 16:06:40.950701 12730 webserver.cc:460] Webserver started at http://127.3.214.195:40223/ using document root <none> and password file <none>
I20260924 16:06:40.951690 12730 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:40.951839 12730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:40.952239 12730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:06:40.956270 12730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "4c94f00ee02f46a98f17f2ecde39e90b"
format_stamp: "Formatted at 2026-09-24 16:06:40 on dist-test-slave-1gc3"
server_key: "b84d9c0055e215048d0dedc302064437"
server_key_iv: "56e6f27d2d473a155283df59a81291be"
server_key_version: "encryptionkey@0"
I20260924 16:06:40.957279 12730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "4c94f00ee02f46a98f17f2ecde39e90b"
format_stamp: "Formatted at 2026-09-24 16:06:40 on dist-test-slave-1gc3"
server_key: "b84d9c0055e215048d0dedc302064437"
server_key_iv: "56e6f27d2d473a155283df59a81291be"
server_key_version: "encryptionkey@0"
I20260924 16:06:40.964530 12730 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260924 16:06:40.969208 12745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:40.971375 12730 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:06:40.971621 12730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "4c94f00ee02f46a98f17f2ecde39e90b"
format_stamp: "Formatted at 2026-09-24 16:06:40 on dist-test-slave-1gc3"
server_key: "b84d9c0055e215048d0dedc302064437"
server_key_iv: "56e6f27d2d473a155283df59a81291be"
server_key_version: "encryptionkey@0"
I20260924 16:06:40.971899 12730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:41.004156 12730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:41.010289 12730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:41.010637 12730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:41.011951 12730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:41.013756 12730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:41.013846 12730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:41.013979 12730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:41.014045 12730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:41.053205 12730 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:34035
I20260924 16:06:41.053282 12858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:34035 every 8 connection(s)
I20260924 16:06:41.054963 12730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:06:41.062781 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 12730
I20260924 16:06:41.062999 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:06:41.063522 3931 external_mini_cluster.cc:1428] Setting key 9267b62a7fc83f2ea727c7e9282c6e1d
Sep 24 16:06:41 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266000, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:06:41.083593 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:41.058402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:38905 (local address 127.3.214.254:44483)
0924 16:06:41.058931 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:41.058938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:41.061252 (+ 2314us) server_negotiation.cc:407] Connection header received
0924 16:06:41.063622 (+ 2370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:41.063630 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:41.063774 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:41.063957 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:41.066042 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:41.067091 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:41.069350 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:41.069881 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:41.070006 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:41.075181 (+ 5175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:41.075190 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:41.075195 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:41.075248 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:41.079189 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:41.080454 (+ 1265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:41.080460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:41.080463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:41.080646 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:41.081210 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:41.081213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:41.081215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:41.081542 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:41.081665 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:41.082812 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0924 16:06:41.083125 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":260,"threads_started":1}
I20260924 16:06:41.084872 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:41.059121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.195:38905)
0924 16:06:41.060464 (+ 1343us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:41.060561 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:41.063218 (+ 2657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:41.064278 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:41.064291 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:41.065070 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:41.065830 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:41.065853 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:41.067291 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:41.067297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:41.069130 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:41.069152 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:41.070179 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:41.070187 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:41.070552 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:41.071462 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:41.071497 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:41.074837 (+ 3340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:06:41.079409 (+ 4572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:41.079423 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:41.079451 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:41.080241 (+ 790us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:41.080781 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:06:41.080787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:06:41.080791 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:06:41.081060 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:06:41.081708 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:06:41.081716 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:06:41.082317 (+ 601us) client_negotiation.cc:769] Sending connection context
0924 16:06:41.082666 (+ 349us) client_negotiation.cc:240] Negotiation successful
0924 16:06:41.083403 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"thread_start_us":524,"threads_started":1}
I20260924 16:06:41.086794 12859 heartbeater.cc:347] Connected to a master server at 127.3.214.254:44483
I20260924 16:06:41.087167 12859 heartbeater.cc:464] Registering TS with master...
I20260924 16:06:41.088074 12859 heartbeater.cc:511] Master 127.3.214.254:44483 requested a full tablet report, sending...
I20260924 16:06:41.089874 12399 ts_manager.cc:194] Registered new tserver with Master: 4c94f00ee02f46a98f17f2ecde39e90b (127.3.214.195:34035)
I20260924 16:06:41.091398 12399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:38905
I20260924 16:06:41.093633 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.214.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.214.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.214.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.193.keytab.
Entry for principal oryx/127.3.214.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.193.keytab.
WARNING: no policy specified for oryx/127.3.214.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.214.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.214.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.194.keytab.
Entry for principal oryx/127.3.214.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.194.keytab.
WARNING: no policy specified for oryx/127.3.214.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.214.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.214.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.195.keytab.
Entry for principal oryx/127.3.214.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.195.keytab.
I20260924 16:06:41.266198 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 12460
Sep 24 16:06:41 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.193@KRBTEST.COM
I20260924 16:06:41.287992 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:41.267623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56042 (local address 127.3.214.193:34195)
0924 16:06:41.268851 (+ 1228us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:41.268862 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:41.268897 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:06:41.268986 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:41.268995 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:41.269292 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:41.269480 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:41.271208 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:41.272397 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:41.273830 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:41.274648 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:41.274790 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:41.280344 (+ 5554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:41.280372 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:41.280398 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:41.280463 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:41.284194 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:41.285341 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:41.285353 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:41.285357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:41.285569 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:41.286057 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:41.286060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:41.286063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:41.286617 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:41.286931 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:41.287249 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 16:06:41.287530 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1029,"thread_start_us":799,"threads_started":1}
I20260924 16:06:41.289788 12584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:41.386592 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 12460
I20260924 16:06:41.407083 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 12595
Sep 24 16:06:41 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.194@KRBTEST.COM
I20260924 16:06:41.426890 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:41.408226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56276 (local address 127.3.214.194:42951)
0924 16:06:41.408891 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:41.408900 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:41.408928 (+ 28us) server_negotiation.cc:407] Connection header received
0924 16:06:41.409007 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:41.409014 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:41.409240 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:41.409418 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:41.410811 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:41.411857 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:41.413134 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:41.413820 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:41.413983 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:41.419101 (+ 5118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:41.419127 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:41.419153 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:41.419216 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:41.423050 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:41.424042 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:41.424053 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:41.424057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:41.424233 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:41.424677 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:41.424682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:41.424684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:41.425230 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:41.425601 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:41.425974 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 16:06:41.426398 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":346,"threads_started":1}
I20260924 16:06:41.428989 12719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:41.517546 12724 heartbeater.cc:503] Master 127.3.214.254:44483 was elected leader, sending a full tablet report...
I20260924 16:06:41.519004 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 12595
W20260924 16:06:41.536769 12385 connection.cc:535] server connection from 127.3.214.194:54951 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 16:06:41.537195 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 12730
Sep 24 16:06:41 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790265998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.195@KRBTEST.COM
I20260924 16:06:41.557076 12874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:41.538439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48084 (local address 127.3.214.195:34035)
0924 16:06:41.539137 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:41.539145 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:41.539177 (+ 32us) server_negotiation.cc:407] Connection header received
0924 16:06:41.539244 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:41.539251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:41.539484 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:41.539653 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:41.541062 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:41.542083 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:41.543487 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:41.544144 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:41.544277 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:41.549978 (+ 5701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:41.550000 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:41.550021 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:41.550070 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:41.553718 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:41.554692 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:41.554702 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:41.554704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:41.554856 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:41.555274 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:41.555278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:41.555280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:41.555708 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:41.556026 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:41.556364 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 16:06:41.556676 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":333,"threads_started":1}
I20260924 16:06:41.559051 12854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:41.652642 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 12730
I20260924 16:06:41.672148 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 12367
I20260924 16:06:41.673905 12429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:41.785312 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 12367
I20260924 16:06:41.809458 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:44483
--webserver_interface=127.3.214.254
--webserver_port=45433
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:44483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:06:42.223489 12877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:42.223912 12877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:42.224045 12877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:42.232447 12877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:06:42.232611 12877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:42.232721 12877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:42.232822 12877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:06:42.232892 12877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:06:42.244508 12877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:44483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:44483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=45433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:42.246485 12877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:42.248283 12877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:42.262633 12883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:42.262650 12885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:42.263155 12882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:42.263609 12877 server_base.cc:1034] running on GCE node
I20260924 16:06:42.264650 12877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:42.266625 12877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:42.267853 12877 hybrid_clock.cc:648] HybridClock initialized: now 1790266002267774 us; error 71 us; skew 500 ppm
I20260924 16:06:42.268285 12877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:42 dist-test-slave-1gc3 krb5kdc[12354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266002, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:42.272054 12877 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:06:42.274027 12877 webserver.cc:460] Webserver started at http://127.3.214.254:45433/ using document root <none> and password file <none>
I20260924 16:06:42.274955 12877 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:42.275099 12877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:42.282467 12877 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 16:06:42.286501 12892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:42.288360 12877 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:06:42.288594 12877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "065e760ec0924ea28d50eef9c1ae9cdb"
format_stamp: "Formatted at 2026-09-24 16:06:38 on dist-test-slave-1gc3"
server_key: "1af1c683aa8a72f4fda097193b5f9af9"
server_key_iv: "a1063925c7703f7a0ada898b2dde1284"
server_key_version: "encryptionkey@0"
I20260924 16:06:42.289885 12877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:42.318032 12877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:42.323793 12877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:42.324115 12877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:42.347388 12877 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:44483
I20260924 16:06:42.347478 12944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:44483 every 8 connection(s)
I20260924 16:06:42.349313 12877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:06:42.350811 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 12877
I20260924 16:06:42.351850 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:34195
--local_ip_for_outbound_sockets=127.3.214.193
--tserver_master_addrs=127.3.214.254:44483
--webserver_port=42331
--webserver_interface=127.3.214.193
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.214.193
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:06:42.360569 12945 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:06:42.365077 12945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb: Bootstrap starting.
I20260924 16:06:42.386236 12945 log.cc:824] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb: Log is configured to *not* fsync() on all Append() calls
I20260924 16:06:42.396167 12945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:06:42.396719 12945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb: Bootstrap complete.
I20260924 16:06:42.407488 12945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065e760ec0924ea28d50eef9c1ae9cdb" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 44483 } }
I20260924 16:06:42.408455 12945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 065e760ec0924ea28d50eef9c1ae9cdb, State: Initialized, Role: FOLLOWER
I20260924 16:06:42.409186 12945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065e760ec0924ea28d50eef9c1ae9cdb" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 44483 } }
I20260924 16:06:42.409439 12945 raft_consensus.cc:388] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:06:42.409555 12945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:06:42.409727 12945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:06:42.414067 12945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065e760ec0924ea28d50eef9c1ae9cdb" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 44483 } }
I20260924 16:06:42.414746 12945 leader_election.cc:302] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 065e760ec0924ea28d50eef9c1ae9cdb; no voters:
I20260924 16:06:42.415449 12945 leader_election.cc:288] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 16:06:42.415644 12949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [term 2 FOLLOWER]: Leader election won for term 2
I20260924 16:06:42.417191 12949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [term 2 LEADER]: Becoming Leader. State: Replica: 065e760ec0924ea28d50eef9c1ae9cdb, State: Running, Role: LEADER
I20260924 16:06:42.417984 12949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065e760ec0924ea28d50eef9c1ae9cdb" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 44483 } }
I20260924 16:06:42.418287 12945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:06:42.420919 12951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 065e760ec0924ea28d50eef9c1ae9cdb. Latest consensus state: current_term: 2 leader_uuid: "065e760ec0924ea28d50eef9c1ae9cdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065e760ec0924ea28d50eef9c1ae9cdb" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 44483 } } }
I20260924 16:06:42.421479 12951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:42.422358 12950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "065e760ec0924ea28d50eef9c1ae9cdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065e760ec0924ea28d50eef9c1ae9cdb" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 44483 } } }
I20260924 16:06:42.422776 12950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb [sys.catalog]: This master's current role is: LEADER
I20260924 16:06:42.427614 12956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:06:42.435798 12956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:06:42.437660 12956 catalog_manager.cc:1259] Loaded cluster ID: 2d421ce8e260487081926e11db19377f
I20260924 16:06:42.437846 12956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:06:42.443015 12956 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:06:42.444988 12956 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 065e760ec0924ea28d50eef9c1ae9cdb: Loaded TSK: 0
I20260924 16:06:42.447220 12956 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:06:42.781864 12947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:42.782361 12947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:42.782624 12947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:42.791173 12947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:42.791333 12947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:42.791504 12947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:06:42.803082 12947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.193.keytab
--principal=oryx/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:34195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=42331
--tserver_master_addrs=127.3.214.254:44483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:42.805028 12947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:42.806802 12947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:42.826054 12971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:42.826164 12972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:42.827080 12974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:42.827891 12947 server_base.cc:1034] running on GCE node
I20260924 16:06:42.828998 12947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:42.830523 12947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:42.831847 12947 hybrid_clock.cc:648] HybridClock initialized: now 1790266002831740 us; error 71 us; skew 500 ppm
I20260924 16:06:42.832347 12947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:42 dist-test-slave-1gc3 krb5kdc[12354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266002, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:42.836699 12947 init.cc:377] Logged in from keytab as oryx/127.3.214.193@KRBTEST.COM (short username oryx)
I20260924 16:06:42.839026 12947 webserver.cc:460] Webserver started at http://127.3.214.193:42331/ using document root <none> and password file <none>
I20260924 16:06:42.840003 12947 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:42.840165 12947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:42.848343 12947 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 16:06:42.852623 12982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:42.854811 12947 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:06:42.855101 12947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "3e4db2ce4d4b417ebd7419571660195f"
format_stamp: "Formatted at 2026-09-24 16:06:39 on dist-test-slave-1gc3"
server_key: "465f4394810055b81cc635ca99b22871"
server_key_iv: "fcb1ae3e7418e1b78f40c0e6ffd899ac"
server_key_version: "encryptionkey@0"
I20260924 16:06:42.856528 12947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:42.879383 12947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:42.885073 12947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:42.885389 12947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:42.886816 12947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:42.888800 12947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:42.888931 12947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:42.889048 12947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:42.889117 12947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:42.929286 12947 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:34195
I20260924 16:06:42.929324 13095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:34195 every 8 connection(s)
I20260924 16:06:42.931341 12947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:06:42.938699 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 12947
I20260924 16:06:42.939388 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:42951
--local_ip_for_outbound_sockets=127.3.214.194
--tserver_master_addrs=127.3.214.254:44483
--webserver_port=35235
--webserver_interface=127.3.214.194
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.214.194
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:06:42 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:42 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 16:06:43.308027 13100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:43.308355 13100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:43.308476 13100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:43.316543 13100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:43.316720 13100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:43.316892 13100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:06:43.327476 13100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.194.keytab
--principal=oryx/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:42951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=35235
--tserver_master_addrs=127.3.214.254:44483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:43.329041 13100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:06:43.331404 13100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:06:43.347409 13106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:43.350584 13107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:43.351859 13109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:43.353538 13100 server_base.cc:1034] running on GCE node
I20260924 16:06:43.354344 13100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:06:43.355453 13100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:43.356734 13100 hybrid_clock.cc:648] HybridClock initialized: now 1790266003356675 us; error 47 us; skew 500 ppm
I20260924 16:06:43.357102 13100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:43 dist-test-slave-1gc3 krb5kdc[12354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266003, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:43.360874 13100 init.cc:377] Logged in from keytab as oryx/127.3.214.194@KRBTEST.COM (short username oryx)
I20260924 16:06:43.368376 13100 webserver.cc:460] Webserver started at http://127.3.214.194:35235/ using document root <none> and password file <none>
I20260924 16:06:43.369220 13100 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:43.369383 13100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:43.376497 13100 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 16:06:43.380501 13116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:43.382880 13100 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260924 16:06:43.383137 13100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "f731f5d5ee3c4f2591ab8b6c3fb245ed"
format_stamp: "Formatted at 2026-09-24 16:06:40 on dist-test-slave-1gc3"
server_key: "7f7db02cf8568469fbfa197a75ab7b5f"
server_key_iv: "44a7e9601e011652e3ae8c52c839e5a9"
server_key_version: "encryptionkey@0"
I20260924 16:06:43.384266 13100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:43.408179 13100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:43.413264 13100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:43.413569 13100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:43.414923 13100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:43.416679 13100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:43.416791 13100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:43.416909 13100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:43.416980 13100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:43.467999 13100 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:42951
I20260924 16:06:43.468046 13229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:42951 every 8 connection(s)
I20260924 16:06:43.469552 13100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:06:43.479874 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 13100
I20260924 16:06:43.480480 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:34035
--local_ip_for_outbound_sockets=127.3.214.195
--tserver_master_addrs=127.3.214.254:44483
--webserver_port=40223
--webserver_interface=127.3.214.195
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.214.195
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:06:43 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:43 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 16:06:44.070139 13234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:06:44.070799 13234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:06:44.071017 13234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:06:44.083004 13234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:06:44.083200 13234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:06:44.083482 13234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:06:44.101516 13234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/krb5kdc/oryx_127.3.214.195.keytab
--principal=oryx/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:34035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=40223
--tserver_master_addrs=127.3.214.254:44483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:06:44.103688 13234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260924 16:06:44.103941 13099 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.155s user 0.002s sys 0.015s
I20260924 16:06:44.105614 13234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:06:44.106837 13099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:42.935410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.193:41691)
0924 16:06:42.936454 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:42.936535 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:42.938811 (+ 2276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:42.940972 (+ 2161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:42.940982 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:42.941680 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:42.942457 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:42.942481 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:42.944312 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:42.944317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:42.945725 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:42.945743 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:42.947053 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:42.947067 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:42.947375 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:42.948341 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:42.948385 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:44.105243 (+1156858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":415,"threads_started":1}
W20260924 16:06:44.106981 13098 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:42.935230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:41691 (local address 127.3.214.254:44483)
0924 16:06:42.935947 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:42.935961 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:42.936741 (+ 780us) server_negotiation.cc:407] Connection header received
0924 16:06:42.939627 (+ 2886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:42.939654 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:42.940198 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:42.940733 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:42.942755 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:42.944122 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:42.945907 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:42.946747 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:42.946952 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:44.105854 (+1158902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.193:41691: BlockingRecv error: recv got EOF from 127.3.214.193:41691 (error 108)
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":277,"threads_started":1}
W20260924 16:06:44.108692 13096 heartbeater.cc:644] Failed to heartbeat to 127.3.214.254:44483 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.214.254:44483: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Sep 24 16:06:44 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:44 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:44.125936 13099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:44.109565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.193:52401)
0924 16:06:44.109875 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:44.109896 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:44.110051 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:44.110444 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:44.110450 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:44.110902 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:44.111103 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:44.111114 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:44.112303 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:44.112309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:44.113502 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:44.113522 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:44.114421 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:44.114427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:44.114611 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:44.120566 (+ 5955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:44.120595 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:44.125560 (+ 4965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184}
W20260924 16:06:44.126801 13098 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:44.109644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:52401 (local address 127.3.214.254:44483)
0924 16:06:44.109865 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:44.109872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:44.109977 (+ 105us) server_negotiation.cc:407] Connection header received
0924 16:06:44.110222 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:44.110227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:44.110304 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:44.110392 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:44.111276 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:44.112117 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:44.113676 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:44.114212 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:44.114343 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:44.126538 (+ 12195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.193:52401: BlockingRecv error: recv got EOF from 127.3.214.193:52401 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 16:06:44.131520 13241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:06:44.136049 13243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:06:44.137740 13234 server_base.cc:1034] running on GCE node
Sep 24 16:06:44 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 16:06:44.139930 13240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Sep 24 16:06:44 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:44.141876 13234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20260924 16:06:44.143797 13098 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:44.130412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:58449 (local address 127.3.214.254:44483)
0924 16:06:44.130638 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:44.130646 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:44.130671 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:06:44.130980 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:44.130986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:44.131083 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:44.131176 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:44.132529 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:44.133963 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:44.136020 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:44.136679 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:44.136811 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:44.143464 (+ 6653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.193:58449: BlockingRecv error: recv got EOF from 127.3.214.193:58449 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 16:06:44.144490 13099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:44.129833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.193:58449)
0924 16:06:44.130078 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:44.130125 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:44.130767 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:44.131402 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:44.131409 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:44.132047 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:44.132321 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:44.132336 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:44.134211 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:44.134218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:44.135843 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:44.135862 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:44.137205 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:44.137211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:44.137415 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:44.138210 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:44.138232 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:44.144183 (+ 5951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":83}
W20260924 16:06:44.145078 13096 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 16:06:44.146845 13234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:06:44.148310 13234 hybrid_clock.cc:648] HybridClock initialized: now 1790266004148169 us; error 86 us; skew 500 ppm
I20260924 16:06:44.148861 13234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:06:44 dist-test-slave-1gc3 krb5kdc[12354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266004, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:06:44.154330 13234 init.cc:377] Logged in from keytab as oryx/127.3.214.195@KRBTEST.COM (short username oryx)
I20260924 16:06:44.157456 13234 webserver.cc:460] Webserver started at http://127.3.214.195:40223/ using document root <none> and password file <none>
I20260924 16:06:44.158876 13234 fs_manager.cc:359] Metadata directory not provided
I20260924 16:06:44.159106 13234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:06:44.170585 13234 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.006s sys 0.001s
I20260924 16:06:44.175715 13250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:44.178346 13234 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:06:44.178601 13234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "4c94f00ee02f46a98f17f2ecde39e90b"
format_stamp: "Formatted at 2026-09-24 16:06:40 on dist-test-slave-1gc3"
server_key: "b84d9c0055e215048d0dedc302064437"
server_key_iv: "56e6f27d2d473a155283df59a81291be"
server_key_version: "encryptionkey@0"
I20260924 16:06:44.179886 13234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:06:44.210191 13234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:06:44.215372 13234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:06:44.215683 13234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:06:44.216980 13234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:06:44.218760 13234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:06:44.218866 13234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:44.219003 13234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:06:44.219110 13234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:06:44.263290 13234 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:34035
I20260924 16:06:44.263329 13363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:34035 every 8 connection(s)
I20260924 16:06:44.265111 13234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:06:44.266728 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 13234
Sep 24 16:06:44 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:44 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:44.305408 13098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:44.268813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55644 (local address 127.3.214.254:44483)
0924 16:06:44.270415 (+ 1602us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:44.270425 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:44.282297 (+ 11872us) server_negotiation.cc:407] Connection header received
0924 16:06:44.282609 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:44.282616 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:44.282725 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:44.282879 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:44.285671 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:44.286841 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:44.290015 (+ 3174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:44.290677 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:44.290801 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:44.294809 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:44.294825 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:44.294867 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:44.294963 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:44.300106 (+ 5143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:44.301334 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:44.301346 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:44.301360 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:44.301701 (+ 341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:06:44.302271 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:06:44.302277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:06:44.302280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:06:44.303217 (+ 937us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:06:44.303634 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0924 16:06:44.304289 (+ 655us) server_negotiation.cc:299] Negotiation successful
0924 16:06:44.304902 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1347}
W20260924 16:06:44.646742 13233 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.161s user 0.007s sys 0.019s
W20260924 16:06:44.648202 13232 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:43.472547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:53549 (local address 127.3.214.254:44483)
0924 16:06:43.474314 (+ 1767us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:43.474325 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:43.474794 (+ 469us) server_negotiation.cc:407] Connection header received
0924 16:06:43.476858 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:43.476867 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:43.477010 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:43.477247 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:43.479143 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:43.480658 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:43.483180 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:43.483995 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:43.484167 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:44.647849 (+1163682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.194:53549: BlockingRecv error: recv got EOF from 127.3.214.194:53549 (error 108)
Metrics: {"server-negotiator.queue_time_us":1497,"thread_start_us":1289,"threads_started":1}
I20260924 16:06:44.648698 13233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:43.472948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.194:53549)
0924 16:06:43.474588 (+ 1640us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:43.474647 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:43.476532 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:43.477538 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:43.477547 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:43.478187 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:43.478885 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:43.478906 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:43.480921 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:43.480927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:43.482972 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:43.482990 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:43.484291 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:43.484298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:43.484595 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:43.485612 (+ 1017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:43.485644 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:44.647583 (+1161939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1307,"thread_start_us":1063,"threads_started":1}
W20260924 16:06:44.649825 13230 heartbeater.cc:644] Failed to heartbeat to 127.3.214.254:44483 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.214.254:44483: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Sep 24 16:06:44 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:44 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:44.664489 13233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:44.650879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.194:56017)
0924 16:06:44.651184 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:44.651208 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:44.651391 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:44.651960 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:44.651964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:44.652338 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:44.652565 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:44.652575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:44.654518 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:44.654525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:44.655742 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:44.655763 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:44.656683 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:44.656693 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:44.656991 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:44.657857 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:44.657887 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:44.663994 (+ 6107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100}
W20260924 16:06:44.664615 13232 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:44.651007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:56017 (local address 127.3.214.254:44483)
0924 16:06:44.651454 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:44.651465 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:44.651498 (+ 33us) server_negotiation.cc:407] Connection header received
0924 16:06:44.651598 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:44.651606 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:44.651726 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:44.651879 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:44.652728 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:44.654223 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:44.655911 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:44.656374 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:44.656513 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:44.664311 (+ 7798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.194:56017: BlockingRecv error: recv got EOF from 127.3.214.194:56017 (error 108)
Metrics: {"server-negotiator.queue_time_us":237}
Sep 24 16:06:44 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:44 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:44.678762 13233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:44.665738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.194:48631)
0924 16:06:44.666001 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:44.666027 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:44.666205 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:44.666804 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:44.666808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:44.667181 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:44.667426 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:44.667438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:44.669377 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:44.669386 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:44.671249 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:44.671274 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:44.672226 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:44.672233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:44.672481 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:44.673315 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:44.673346 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:44.678424 (+ 5078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94}
W20260924 16:06:44.679165 13230 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:06:44.678967 13232 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:44.665956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:48631 (local address 127.3.214.254:44483)
0924 16:06:44.666278 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:44.666287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:44.666321 (+ 34us) server_negotiation.cc:407] Connection header received
0924 16:06:44.666434 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:44.666441 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:44.666582 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:44.666730 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:44.667600 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:44.669083 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:44.671482 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:44.672026 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:44.672154 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:44.678705 (+ 6551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.194:48631: BlockingRecv error: recv got EOF from 127.3.214.194:48631 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 24 16:06:45 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:45 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:45.160156 13370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:45.145959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.193:43007)
0924 16:06:45.147199 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:45.147231 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:45.147456 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:45.148000 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:45.148006 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:45.148812 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:45.149120 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:45.149136 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:45.150340 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:45.150346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:45.152260 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:45.152283 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:45.153379 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:45.153387 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:45.153632 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:45.154538 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:45.154568 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:45.159776 (+ 5208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1035,"thread_start_us":814,"threads_started":1}
W20260924 16:06:45.160567 13232 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:45.146335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:43007 (local address 127.3.214.254:44483)
0924 16:06:45.146565 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:45.146572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:45.147363 (+ 791us) server_negotiation.cc:407] Connection header received
0924 16:06:45.147668 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:45.147673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:45.147773 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:45.147890 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:45.149344 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:45.150164 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:45.152579 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:45.153104 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:45.153251 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:45.160208 (+ 6957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.193:43007: BlockingRecv error: recv got EOF from 127.3.214.193:43007 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260924 16:06:45.325067 13368 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.032s user 0.009s sys 0.007s
W20260924 16:06:45.326606 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:44.268950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:34871 (local address 127.3.214.254:44483)
0924 16:06:44.274584 (+ 5634us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:44.274593 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:44.281251 (+ 6658us) server_negotiation.cc:407] Connection header received
0924 16:06:44.284718 (+ 3467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:44.284724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:44.284823 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:44.284931 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:44.287473 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:44.288348 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:44.290590 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:44.291530 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:44.291756 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:45.326309 (+1034553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.195:34871: BlockingRecv error: recv got EOF from 127.3.214.195:34871 (error 108)
Metrics: {"server-negotiator.queue_time_us":5447,"thread_start_us":5289,"threads_started":1}
I20260924 16:06:45.327234 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:44.269970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.195:34871)
0924 16:06:44.280976 (+ 11006us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:44.281043 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:44.284260 (+ 3217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:44.285331 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:44.285343 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:44.286304 (+ 961us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:44.287277 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:44.287303 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:44.288502 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:44.288506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:44.290283 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:44.290319 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:44.291899 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:44.291910 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:44.292297 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:44.293292 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:44.293330 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:45.325991 (+1032661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":10591,"thread_start_us":10760,"threads_started":1}
W20260924 16:06:45.328410 13364 heartbeater.cc:644] Failed to heartbeat to 127.3.214.254:44483 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.214.254:44483: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Sep 24 16:06:45 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:45 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:45.340389 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:45.329198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.195:37535)
0924 16:06:45.329460 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:45.329488 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:45.329669 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:45.330025 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:45.330029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:45.330512 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:45.330742 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:45.330753 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:45.331931 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:45.331936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:45.333094 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:45.333114 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:45.334271 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:45.334279 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:45.334489 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:45.335271 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:45.335291 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:45.340033 (+ 4742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":82}
W20260924 16:06:45.340608 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:45.329337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:37535 (local address 127.3.214.254:44483)
0924 16:06:45.329698 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:45.329703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:45.329727 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:06:45.329790 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:45.329794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:45.329862 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:45.329973 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:45.330954 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:45.331739 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:45.333372 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:45.333960 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:45.334148 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:45.340379 (+ 6231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.195:37535: BlockingRecv error: recv got EOF from 127.3.214.195:37535 (error 108)
Metrics: {"server-negotiator.queue_time_us":217}
Sep 24 16:06:45 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:45 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:45.352155 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:45.341474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.195:34591)
0924 16:06:45.341687 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:45.341709 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:45.341844 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:45.342217 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:45.342221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:45.342595 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:45.342824 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:45.342835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:45.344230 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:45.344235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:45.345398 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:45.345416 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:45.346413 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:45.346420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:45.346651 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:45.347434 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:45.347460 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:45.351892 (+ 4432us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260924 16:06:45.352650 13364 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:06:45.353083 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:45.341545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:34591 (local address 127.3.214.254:44483)
0924 16:06:45.341702 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:45.341707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:45.341783 (+ 76us) server_negotiation.cc:407] Connection header received
0924 16:06:45.341961 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:45.341965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:45.342032 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:45.342171 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:45.343042 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:45.344019 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:45.345583 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:45.346180 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:45.346352 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:45.352213 (+ 5861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.195:34591: BlockingRecv error: recv got EOF from 127.3.214.195:34591 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 16:06:45 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:45 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:45.691206 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:45.679912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.194:46215)
0924 16:06:45.680671 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:45.680691 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:45.680825 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:45.681441 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:45.681445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:45.682062 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:45.682310 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:45.682321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:45.683726 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:45.683733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:45.684868 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:45.684886 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:45.685972 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:45.685978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:45.686187 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:45.686837 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:45.686856 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:45.690939 (+ 4083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":373,"threads_started":1}
W20260924 16:06:45.691591 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:45.680038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:46215 (local address 127.3.214.254:44483)
0924 16:06:45.680284 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:45.680290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:45.680806 (+ 516us) server_negotiation.cc:407] Connection header received
0924 16:06:45.681093 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:45.681098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:45.681194 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:45.681321 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:45.682497 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:45.683391 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:45.685141 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:45.685693 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:45.685855 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:45.691283 (+ 5428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.194:46215: BlockingRecv error: recv got EOF from 127.3.214.194:46215 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 24 16:06:46 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:46 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:46.172973 13372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:46.162008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.193:36587)
0924 16:06:46.162814 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:46.162837 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:46.163050 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:46.163464 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:46.163469 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:46.164143 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:46.164400 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:46.164416 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:46.165608 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:46.165614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:46.166662 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:46.166680 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:46.167490 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:46.167497 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:46.167681 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:46.168401 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:46.168422 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:46.172603 (+ 4181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":402,"threads_started":1}
W20260924 16:06:46.173189 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:46.162275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:36587 (local address 127.3.214.254:44483)
0924 16:06:46.162569 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:46.162577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:46.162990 (+ 413us) server_negotiation.cc:407] Connection header received
0924 16:06:46.163201 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:46.163206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:46.163286 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:46.163398 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:46.164607 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:46.165438 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:46.166837 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:46.167306 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:46.167430 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:46.172916 (+ 5486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.193:36587: BlockingRecv error: recv got EOF from 127.3.214.193:36587 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 24 16:06:46 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:46 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:46.363881 13373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:46.353490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.195:51051)
0924 16:06:46.354201 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:46.354218 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:46.354367 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:46.354813 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:46.354816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:46.355280 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:46.355518 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:46.355532 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:46.356755 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:46.356759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:46.357763 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:46.357777 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:46.358643 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:46.358648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:46.358790 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:46.359484 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:46.359504 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:46.363621 (+ 4117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":411,"threads_started":1}
W20260924 16:06:46.364250 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:46.353620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:51051 (local address 127.3.214.254:44483)
0924 16:06:46.353873 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:46.353881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:46.354301 (+ 420us) server_negotiation.cc:407] Connection header received
0924 16:06:46.354504 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:46.354510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:46.354628 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:46.354772 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:46.355727 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:46.356550 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:46.357925 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:46.358459 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:46.358564 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:46.363956 (+ 5392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.195:51051: BlockingRecv error: recv got EOF from 127.3.214.195:51051 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 16:06:46 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:46 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:46.705189 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:46.693015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.194:56575)
0924 16:06:46.693641 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:46.693658 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:46.693799 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:46.694354 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:46.694358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:46.694828 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:46.695094 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:46.695105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:46.696407 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:46.696428 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:46.697466 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:46.697484 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:46.698458 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:46.698472 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:46.698631 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:46.699542 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:46.699565 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:46.704725 (+ 5160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":369,"threads_started":1}
W20260924 16:06:46.705338 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:46.693140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:56575 (local address 127.3.214.254:44483)
0924 16:06:46.693372 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:46.693377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:46.693740 (+ 363us) server_negotiation.cc:407] Connection header received
0924 16:06:46.694010 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:46.694015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:46.694153 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:46.694271 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:46.695244 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:46.696192 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:46.697670 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:46.698239 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:46.698385 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:46.705035 (+ 6650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.194:56575: BlockingRecv error: recv got EOF from 127.3.214.194:56575 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 24 16:06:47 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:47 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:47.185142 13375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:47.174382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.193:37725)
0924 16:06:47.175036 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:47.175058 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:47.175240 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:47.175756 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:47.175762 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:47.176252 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:47.176500 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:47.176511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:47.177649 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:47.177658 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:47.178705 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:47.178720 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:47.179437 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:47.179442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:47.179633 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:47.180335 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:47.180356 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:47.184847 (+ 4491us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":351,"threads_started":1}
W20260924 16:06:47.185516 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:47.174592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:37725 (local address 127.3.214.254:44483)
0924 16:06:47.174848 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:47.174854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:47.175168 (+ 314us) server_negotiation.cc:407] Connection header received
0924 16:06:47.175387 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:47.175393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:47.175492 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:47.175617 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:47.176696 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:47.177476 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:47.178850 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:47.179279 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:47.179387 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:47.185195 (+ 5808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.193:37725: BlockingRecv error: recv got EOF from 127.3.214.193:37725 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 24 16:06:47 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:47 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:47.376082 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:47.365094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.195:54803)
0924 16:06:47.365700 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:47.365719 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:47.365874 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:47.366305 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:47.366310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:47.366847 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:47.367087 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:47.367098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:47.368254 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:47.368258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:47.369499 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:47.369513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:47.370477 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:47.370483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:47.370690 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:47.371464 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:47.371491 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:47.375761 (+ 4270us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":327,"threads_started":1}
W20260924 16:06:47.376271 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:47.365212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:54803 (local address 127.3.214.254:44483)
0924 16:06:47.365455 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:47.365459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:47.365813 (+ 354us) server_negotiation.cc:407] Connection header received
0924 16:06:47.366007 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:47.366011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:47.366112 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:47.366224 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:47.367267 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:47.368091 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:47.369665 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:47.370297 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:47.370441 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:47.376012 (+ 5571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.195:54803: BlockingRecv error: recv got EOF from 127.3.214.195:54803 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 24 16:06:47 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:47 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:47.717928 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:47.706575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.194:36177)
0924 16:06:47.707177 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:47.707197 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:47.707364 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:47.707886 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:47.707890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:47.708298 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:47.708531 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:47.708541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:47.709814 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:47.709819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:47.710844 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:47.710859 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:47.711871 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:47.711877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:47.712038 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:47.712738 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:47.712757 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:47.717545 (+ 4788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":305,"threads_started":1}
W20260924 16:06:47.718138 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:47.706772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:36177 (local address 127.3.214.254:44483)
0924 16:06:47.707088 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:47.707096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:47.707303 (+ 207us) server_negotiation.cc:407] Connection header received
0924 16:06:47.707546 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:47.707587 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:47.707688 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:47.707804 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:47.708725 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:47.709634 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:47.711044 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:47.711655 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:47.711791 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:47.717885 (+ 6094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.194:36177: BlockingRecv error: recv got EOF from 127.3.214.194:36177 (error 108)
Metrics: {"server-negotiator.queue_time_us":141}
Sep 24 16:06:48 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:48 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:48.197310 13378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:48.186553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.193:38513)
0924 16:06:48.187209 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:48.187227 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:48.187401 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:48.187771 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:48.187776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:48.188229 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:48.188423 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:48.188434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:48.189532 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:48.189537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:48.190694 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:48.190709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:48.191586 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:48.191592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:48.191766 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:48.192552 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:48.192573 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:48.196948 (+ 4375us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":373,"threads_started":1}
W20260924 16:06:48.197522 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:48.186674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:38513 (local address 127.3.214.254:44483)
0924 16:06:48.186898 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:48.186905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:48.187344 (+ 439us) server_negotiation.cc:407] Connection header received
0924 16:06:48.187537 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:48.187543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:48.187613 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:48.187702 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:48.188589 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:48.189372 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:48.190849 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:48.191413 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:48.191539 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:48.197250 (+ 5711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.193:38513: BlockingRecv error: recv got EOF from 127.3.214.193:38513 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 16:06:48 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:48 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:48.388245 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:48.377391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.195:33175)
0924 16:06:48.378088 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:48.378143 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:48.378327 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:48.378713 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:48.378718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:48.379136 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:48.379350 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:48.379360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:48.380409 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:48.380414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:48.381370 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:48.381385 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:48.382171 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:48.382177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:48.382355 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:48.383068 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:48.383088 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:48.387942 (+ 4854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":355,"threads_started":1}
W20260924 16:06:48.388446 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:48.377572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:33175 (local address 127.3.214.254:44483)
0924 16:06:48.377847 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:48.377854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:48.378239 (+ 385us) server_negotiation.cc:407] Connection header received
0924 16:06:48.378454 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:48.378460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:48.378550 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:48.378636 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:48.379511 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:48.380265 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:48.381519 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:48.381979 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:48.382096 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:48.388228 (+ 6132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.195:33175: BlockingRecv error: recv got EOF from 127.3.214.195:33175 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 24 16:06:48 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:48 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:48.730847 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:48.719252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.194:36595)
0924 16:06:48.720024 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:48.720044 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:48.720203 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:48.720554 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:48.720558 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:48.721064 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:48.721300 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:48.721311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:48.722525 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:48.722532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:48.723791 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:48.723810 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:48.724711 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:48.724716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:48.724889 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:48.725804 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:48.725828 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:48.730444 (+ 4616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":459,"threads_started":1}
W20260924 16:06:48.731104 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:48.719262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:36595 (local address 127.3.214.254:44483)
0924 16:06:48.719463 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:48.719468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:48.720148 (+ 680us) server_negotiation.cc:407] Connection header received
0924 16:06:48.720316 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:48.720319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:48.720390 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:48.720487 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:48.721467 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:48.722296 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:48.724004 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:48.724503 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:48.724623 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:48.730764 (+ 6141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.194:36595: BlockingRecv error: recv got EOF from 127.3.214.194:36595 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 16:06:49 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:49 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:49.213515 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:49.199006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.193:42375)
0924 16:06:49.199576 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:49.199599 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:49.199752 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:49.200217 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:49.200221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:49.200757 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:49.200946 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:49.200959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:49.202318 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:49.202323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:49.203347 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:49.203363 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:49.204303 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:49.204308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:49.204510 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:49.208607 (+ 4097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:49.208634 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:49.213222 (+ 4588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":303,"threads_started":1}
W20260924 16:06:49.213862 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:49.199200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:42375 (local address 127.3.214.254:44483)
0924 16:06:49.199433 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:49.199441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:49.199702 (+ 261us) server_negotiation.cc:407] Connection header received
0924 16:06:49.199943 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:49.199949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:49.200055 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:49.200189 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:49.201131 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:49.202145 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:49.203532 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:49.204103 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:49.204230 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:49.213563 (+ 9333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.193:42375: BlockingRecv error: recv got EOF from 127.3.214.193:42375 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 16:06:49 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:49 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:49.401315 13382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:49.389593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.195:40693)
0924 16:06:49.390321 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:49.390342 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:49.390519 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:49.391002 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:49.391006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:49.391522 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:49.391797 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:49.391811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:49.393273 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:49.393278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:49.394779 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:49.394798 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:49.395820 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:49.395827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:49.396024 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:49.396752 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:49.396773 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:49.401009 (+ 4236us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":384,"threads_started":1}
W20260924 16:06:49.401717 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:49.389823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:40693 (local address 127.3.214.254:44483)
0924 16:06:49.390185 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:49.390193 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:49.390511 (+ 318us) server_negotiation.cc:407] Connection header received
0924 16:06:49.390663 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:49.390668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:49.390784 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:49.390908 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:49.392040 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:49.393047 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:49.394984 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:49.395634 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:49.395814 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:49.401355 (+ 5541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.195:40693: BlockingRecv error: recv got EOF from 127.3.214.195:40693 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 24 16:06:49 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:49 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:49.743278 13383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:49.732192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.194:57465)
0924 16:06:49.732903 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:49.732927 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:49.733132 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:49.733650 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:49.733656 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:49.734293 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:49.734568 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:49.734582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:49.735871 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:49.735876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:49.736986 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:49.737001 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:49.737822 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:49.737826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:49.737976 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:49.738701 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:49.738723 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:49.742984 (+ 4261us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":361,"threads_started":1}
W20260924 16:06:49.743623 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:49.732572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:57465 (local address 127.3.214.254:44483)
0924 16:06:49.732831 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:49.732840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:49.733099 (+ 259us) server_negotiation.cc:407] Connection header received
0924 16:06:49.733314 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:49.733322 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:49.733434 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:49.733536 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:49.734749 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:49.735668 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:49.737179 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:49.737654 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:49.737776 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:49.743328 (+ 5552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.194:57465: BlockingRecv error: recv got EOF from 127.3.214.194:57465 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 16:06:50 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:50 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:50.225231 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:50.214864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.193:55479)
0924 16:06:50.215436 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:50.215457 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:50.215591 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:50.216119 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:50.216123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:50.216575 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:50.216762 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:50.216772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:50.218186 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:50.218191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:50.219181 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:50.219194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:50.220047 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:50.220051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:50.220182 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:50.220815 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:50.220834 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:50.224951 (+ 4117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":295,"threads_started":1}
W20260924 16:06:50.225589 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:50.214963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:55479 (local address 127.3.214.254:44483)
0924 16:06:50.215229 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:50.215237 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:50.215547 (+ 310us) server_negotiation.cc:407] Connection header received
0924 16:06:50.215822 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:50.215828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:50.215923 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:50.216046 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:50.216941 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:50.217945 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:50.219343 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:50.219888 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:50.220034 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:50.225255 (+ 5221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.193:55479: BlockingRecv error: recv got EOF from 127.3.214.193:55479 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 24 16:06:50 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:50 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:50.417994 13385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:50.402601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.195:36149)
0924 16:06:50.403287 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:50.403308 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:50.403500 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:50.403898 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:50.403902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:50.404441 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:50.404638 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:50.404651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:50.405850 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:50.405854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:50.407024 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:50.407041 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:50.407846 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:50.407851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:50.408047 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:50.413062 (+ 5015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:50.413092 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:50.417692 (+ 4600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":349,"threads_started":1}
W20260924 16:06:50.418406 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:50.402833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:36149 (local address 127.3.214.254:44483)
0924 16:06:50.403125 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:50.403158 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:50.403415 (+ 257us) server_negotiation.cc:407] Connection header received
0924 16:06:50.403641 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:50.403646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:50.403733 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:50.403829 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:50.404814 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:50.405691 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:50.407204 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:50.407676 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:50.407806 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:50.418049 (+ 10243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.195:36149: BlockingRecv error: recv got EOF from 127.3.214.195:36149 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 24 16:06:50 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:50 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:50.756224 13386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:50.744501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.194:50521)
0924 16:06:50.745682 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:50.745702 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:50.745858 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:50.746300 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:50.746305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:50.746918 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:50.747202 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:50.747214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:50.748362 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:50.748367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:50.749357 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:50.749370 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:50.750215 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:50.750221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:50.750399 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:50.751131 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:50.751151 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:50.755913 (+ 4762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1048,"thread_start_us":848,"threads_started":1}
W20260924 16:06:50.756788 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:50.744774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:50521 (local address 127.3.214.254:44483)
0924 16:06:50.745154 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:50.745162 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:50.745797 (+ 635us) server_negotiation.cc:407] Connection header received
0924 16:06:50.745996 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:50.746000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:50.746072 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:50.746205 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:50.747359 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:50.748204 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:50.749495 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:50.750000 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:50.750216 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:50.756249 (+ 6033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.194:50521: BlockingRecv error: recv got EOF from 127.3.214.194:50521 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Sep 24 16:06:51 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:51 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:51.238077 13387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:51.226901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.193:33351)
0924 16:06:51.227652 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:51.227672 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:51.227853 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:51.228230 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:51.228236 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:51.228790 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:51.229045 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:51.229062 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:51.230338 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:51.230343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:51.231392 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:51.231407 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:51.232208 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:51.232212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:51.232345 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:51.233166 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:51.233194 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:51.237780 (+ 4586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":354,"threads_started":1}
W20260924 16:06:51.238372 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:51.227064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:33351 (local address 127.3.214.254:44483)
0924 16:06:51.227344 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:51.227353 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:51.227797 (+ 444us) server_negotiation.cc:407] Connection header received
0924 16:06:51.227993 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:51.227999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:51.228082 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:51.228175 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:51.229271 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:51.230154 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:51.231584 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:51.232051 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:51.232167 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:51.238045 (+ 5878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.193:33351: BlockingRecv error: recv got EOF from 127.3.214.193:33351 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 24 16:06:51 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:51 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:51.430366 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:51.419293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.195:39583)
0924 16:06:51.419884 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:51.419903 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:51.420058 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:51.420500 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:51.420504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:51.421037 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:51.421254 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:51.421266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:51.422460 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:51.422464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:51.423557 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:51.423575 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:51.424451 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:51.424457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:51.424684 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:51.425452 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:51.425472 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:51.429947 (+ 4475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":323,"threads_started":1}
W20260924 16:06:51.430516 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:51.419429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:39583 (local address 127.3.214.254:44483)
0924 16:06:51.419707 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:51.419713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:51.420028 (+ 315us) server_negotiation.cc:407] Connection header received
0924 16:06:51.420207 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:51.420211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:51.420302 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:51.420433 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:51.421435 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:51.422264 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:51.423746 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:51.424231 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:51.424352 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:51.430244 (+ 5892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.195:39583: BlockingRecv error: recv got EOF from 127.3.214.195:39583 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 24 16:06:51 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:51 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:51.767952 13389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:51.757542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.194:36305)
0924 16:06:51.758233 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:51.758253 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:51.758393 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:51.758871 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:51.758875 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:51.759325 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:51.759522 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:51.759531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:51.760795 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:51.760799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:51.761837 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:51.761850 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:51.762862 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:51.762868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:51.763054 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:51.763759 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:51.763788 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:51.767668 (+ 3880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":322,"threads_started":1}
W20260924 16:06:51.768240 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:51.757711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:36305 (local address 127.3.214.254:44483)
0924 16:06:51.758000 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:51.758006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:51.758345 (+ 339us) server_negotiation.cc:407] Connection header received
0924 16:06:51.758555 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:51.758560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:51.758650 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:51.758781 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:51.759706 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:51.760651 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:51.762002 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:51.762640 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:51.762785 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:51.767977 (+ 5192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.194:36305: BlockingRecv error: recv got EOF from 127.3.214.194:36305 (error 108)
Metrics: {"server-negotiator.queue_time_us":151}
Sep 24 16:06:52 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:52 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:52.250092 13390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:52.239375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.193:60421)
0924 16:06:52.240025 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:52.240045 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:52.240206 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:52.240603 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:52.240607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:52.241081 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:52.241310 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:52.241321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:52.242607 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:52.242612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:52.243762 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:52.243781 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:52.244641 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:52.244646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:52.244786 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:52.245594 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:52.245616 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:52.249774 (+ 4158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":328,"threads_started":1}
W20260924 16:06:52.250501 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:52.239530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:60421 (local address 127.3.214.254:44483)
0924 16:06:52.239803 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:52.239810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:52.240174 (+ 364us) server_negotiation.cc:407] Connection header received
0924 16:06:52.240349 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:52.240353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:52.240440 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:52.240546 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:52.241504 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:52.242414 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:52.243940 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:52.244440 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:52.244585 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:52.250177 (+ 5592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.193:60421: BlockingRecv error: recv got EOF from 127.3.214.193:60421 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 24 16:06:52 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:52 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:52.442713 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:52.431619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.195:58509)
0924 16:06:52.432338 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:52.432362 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:52.432537 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:52.433021 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:52.433026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:52.433503 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:52.433717 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:52.433728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:52.434919 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:52.434924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:52.435960 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:52.435976 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:52.436873 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:52.436878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:52.437032 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:52.437829 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:52.437859 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:52.442314 (+ 4455us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":295,"threads_started":1}
W20260924 16:06:52.442853 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:52.431797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:58509 (local address 127.3.214.254:44483)
0924 16:06:52.432052 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:52.432060 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:52.432492 (+ 432us) server_negotiation.cc:407] Connection header received
0924 16:06:52.432677 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:52.432683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:52.432812 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:52.432910 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:52.433871 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:52.434731 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:52.436139 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:52.436665 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:52.436815 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:52.442569 (+ 5754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.195:58509: BlockingRecv error: recv got EOF from 127.3.214.195:58509 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 24 16:06:52 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:52 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:52.779892 13392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:52.768984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.194:46357)
0924 16:06:52.769687 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:52.769708 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:52.769880 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:52.770283 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:52.770288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:52.770817 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:52.771015 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:52.771025 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:52.772362 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:52.772367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:52.773382 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:52.773394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:52.774298 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:52.774305 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:52.774494 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:52.775220 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:52.775239 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:52.779485 (+ 4246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":331,"threads_started":1}
W20260924 16:06:52.780069 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:52.769116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:46357 (local address 127.3.214.254:44483)
0924 16:06:52.769401 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:52.769409 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:52.769806 (+ 397us) server_negotiation.cc:407] Connection header received
0924 16:06:52.770011 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:52.770016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:52.770141 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:52.770241 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:52.771163 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:52.772204 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:52.773535 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:52.774061 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:52.774234 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:52.779790 (+ 5556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.194:46357: BlockingRecv error: recv got EOF from 127.3.214.194:46357 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 24 16:06:53 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:53 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:53.262662 13393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:53.251455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.193:41587)
0924 16:06:53.252115 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:53.252136 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:53.252292 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:53.252774 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:53.252777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:53.253324 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:53.253521 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:53.253531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:53.254782 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:53.254787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:53.255854 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:53.255867 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:53.256733 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:53.256738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:53.256893 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:53.257656 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:53.257679 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:53.262361 (+ 4682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":354,"threads_started":1}
W20260924 16:06:53.262909 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:53.251628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:41587 (local address 127.3.214.254:44483)
0924 16:06:53.251944 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:53.251951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:53.252243 (+ 292us) server_negotiation.cc:407] Connection header received
0924 16:06:53.252498 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:53.252503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:53.252619 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:53.252732 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:53.253702 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:53.254614 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:53.256041 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:53.256531 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:53.256667 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:53.262636 (+ 5969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.193:41587: BlockingRecv error: recv got EOF from 127.3.214.193:41587 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 24 16:06:53 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:53 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:53.455852 13394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:53.444387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.195:40613)
0924 16:06:53.445012 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:53.445031 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:53.445178 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:53.445713 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:53.445719 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:53.446379 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:53.446622 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:53.446637 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:53.448085 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:53.448092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:53.449173 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:53.449194 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:53.450345 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:53.450353 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:53.450538 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:53.451357 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:53.451385 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:53.455556 (+ 4171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":313,"threads_started":1}
W20260924 16:06:53.456305 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:53.444574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:40613 (local address 127.3.214.254:44483)
0924 16:06:53.444803 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:53.444809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:53.445162 (+ 353us) server_negotiation.cc:407] Connection header received
0924 16:06:53.445318 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:53.445323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:53.445425 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:53.445562 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:53.446856 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:53.447785 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:53.449412 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:53.450037 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:53.450216 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:53.455943 (+ 5727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.195:40613: BlockingRecv error: recv got EOF from 127.3.214.195:40613 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 24 16:06:53 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:53 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:53.792897 13395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:53.781215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.194:37051)
0924 16:06:53.781847 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:53.781866 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:53.782005 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:53.782475 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:53.782479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:53.783009 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:53.783222 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:53.783232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:53.784597 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:53.784601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:53.785760 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:53.785777 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:53.786813 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:53.786831 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:53.787031 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:53.787704 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:53.787726 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:53.792597 (+ 4871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":309,"threads_started":1}
W20260924 16:06:53.793460 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:53.781273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:37051 (local address 127.3.214.254:44483)
0924 16:06:53.781532 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:53.781540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:53.781986 (+ 446us) server_negotiation.cc:407] Connection header received
0924 16:06:53.782198 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:53.782203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:53.782287 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:53.782404 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:53.783413 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:53.784402 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:53.785955 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:53.786556 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:53.786715 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:53.792964 (+ 6249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.194:37051: BlockingRecv error: recv got EOF from 127.3.214.194:37051 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 16:06:54 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:54 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:54.280082 13396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:54.263901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.193:38693)
0924 16:06:54.264683 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:54.264710 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:54.264927 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:54.265477 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:54.265484 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:54.266176 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:54.266515 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:54.266533 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:54.267722 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:54.267727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:54.269054 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:54.269073 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:54.270037 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:54.270045 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:54.270266 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:54.275653 (+ 5387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:54.275680 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:54.279711 (+ 4031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":334,"threads_started":1}
W20260924 16:06:54.280277 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:54.264112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:38693 (local address 127.3.214.254:44483)
0924 16:06:54.264363 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:54.264370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:54.264869 (+ 499us) server_negotiation.cc:407] Connection header received
0924 16:06:54.265157 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:54.265163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:54.265237 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:54.265388 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:54.266741 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:54.267536 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:54.269218 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:54.269806 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:54.269946 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:54.280007 (+ 10061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.193:38693: BlockingRecv error: recv got EOF from 127.3.214.193:38693 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 24 16:06:54 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:54 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:54.467846 13397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:54.457415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.195:53077)
0924 16:06:54.458195 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:54.458213 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:54.458377 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:54.458717 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:54.458722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:54.459164 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:54.459321 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:54.459329 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:54.460399 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:54.460405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:54.461377 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:54.461390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:54.462172 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:54.462178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:54.462354 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:54.462936 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:54.462955 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:54.467526 (+ 4571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":459,"threads_started":1}
W20260924 16:06:54.468073 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:54.457641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:53077 (local address 127.3.214.254:44483)
0924 16:06:54.457983 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:54.457991 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:54.458328 (+ 337us) server_negotiation.cc:407] Connection header received
0924 16:06:54.458508 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:54.458512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:54.458581 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:54.458665 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:54.459481 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:54.460248 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:54.461524 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:54.461973 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:54.462092 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:54.467811 (+ 5719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.195:53077: BlockingRecv error: recv got EOF from 127.3.214.195:53077 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 24 16:06:54 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:54 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:54.805747 13398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:54.794224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.194:40767)
0924 16:06:54.794941 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:54.794971 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:54.795180 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:54.795580 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:54.795585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:54.796091 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:54.796296 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:54.796313 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:54.797437 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:54.797441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:54.798704 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:54.798718 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:54.799674 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:54.799682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:54.799955 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:54.800736 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:54.800757 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:54.805443 (+ 4686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":371,"threads_started":1}
W20260924 16:06:54.806222 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:54.794484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:40767 (local address 127.3.214.254:44483)
0924 16:06:54.794887 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:54.794895 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:54.795080 (+ 185us) server_negotiation.cc:407] Connection header received
0924 16:06:54.795352 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:54.795356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:54.795432 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:54.795524 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:54.796466 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:54.797282 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:54.798852 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:54.799397 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:54.799536 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:54.805842 (+ 6306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.194:40767: BlockingRecv error: recv got EOF from 127.3.214.194:40767 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 24 16:06:55 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:55 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:55.292505 13399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:55.281644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.193:34763)
0924 16:06:55.282286 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:55.282306 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:55.282462 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:55.282813 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:55.282816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:55.283346 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:55.283580 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:55.283591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:55.284889 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:55.284893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:55.286329 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:55.286350 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:55.287234 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:55.287240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:55.287420 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:55.288128 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:55.288148 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:55.292084 (+ 3936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":339,"threads_started":1}
W20260924 16:06:55.292721 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:55.282058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:34763 (local address 127.3.214.254:44483)
0924 16:06:55.282283 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:55.282288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:55.282415 (+ 127us) server_negotiation.cc:407] Connection header received
0924 16:06:55.282577 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:55.282581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:55.282654 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:55.282774 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:55.283813 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:55.284680 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:55.286518 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:55.287057 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:55.287190 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:55.292407 (+ 5217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.193:34763: BlockingRecv error: recv got EOF from 127.3.214.193:34763 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 24 16:06:55 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:55 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:55.480592 13400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:55.469420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.195:37689)
0924 16:06:55.470166 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:55.470188 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:55.470362 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:55.470821 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:55.470827 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:55.471422 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:55.471718 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:55.471732 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:55.473183 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:55.473188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:55.474418 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:55.474435 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:55.475269 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:55.475275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:55.475435 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:55.476196 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:55.476220 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:55.480278 (+ 4058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":400,"threads_started":1}
W20260924 16:06:55.480932 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:55.469606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:37689 (local address 127.3.214.254:44483)
0924 16:06:55.469834 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:55.469841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:55.470309 (+ 468us) server_negotiation.cc:407] Connection header received
0924 16:06:55.470506 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:55.470512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:55.470595 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:55.470736 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:55.471924 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:55.473032 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:55.474594 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:55.475067 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:55.475185 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:55.480614 (+ 5429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.195:37689: BlockingRecv error: recv got EOF from 127.3.214.195:37689 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 24 16:06:55 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:55 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:55.817793 13401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:55.807134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.194:58155)
0924 16:06:55.807802 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:55.807826 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:55.808018 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:55.808389 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:55.808394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:55.808915 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:55.809156 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:55.809166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:55.810309 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:55.810315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:55.811309 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:55.811321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:55.812135 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:55.812141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:55.812321 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:55.813088 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:55.813109 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:55.817467 (+ 4358us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":349,"threads_started":1}
W20260924 16:06:55.818085 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:55.807413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:58155 (local address 127.3.214.254:44483)
0924 16:06:55.807790 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:55.807798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:55.807935 (+ 137us) server_negotiation.cc:407] Connection header received
0924 16:06:55.808172 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:55.808176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:55.808249 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:55.808342 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:55.809313 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:55.810179 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:55.811460 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:55.811965 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:55.812078 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:55.817791 (+ 5713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.194:58155: BlockingRecv error: recv got EOF from 127.3.214.194:58155 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Sep 24 16:06:56 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:56 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:56.305430 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:56.293939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.193:42963)
0924 16:06:56.294706 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:56.294725 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:56.294895 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:56.295568 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:56.295575 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:56.296121 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:56.296349 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:56.296361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:56.297485 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:56.297489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:56.298532 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:56.298549 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:56.299322 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:56.299328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:56.299504 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:56.300351 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:56.300382 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:56.305028 (+ 4646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":370,"threads_started":1}
W20260924 16:06:56.305567 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:56.294041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:42963 (local address 127.3.214.254:44483)
0924 16:06:56.294295 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:56.294302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:56.294830 (+ 528us) server_negotiation.cc:407] Connection header received
0924 16:06:56.295098 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:56.295111 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:56.295263 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:56.295452 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:56.296529 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:56.297325 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:56.298685 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:56.299123 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:56.299236 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:56.305274 (+ 6038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.193:42963: BlockingRecv error: recv got EOF from 127.3.214.193:42963 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 24 16:06:56 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:56 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:56.492903 13403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:56.481782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.195:45519)
0924 16:06:56.482369 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:56.482388 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:56.482524 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:56.482879 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:56.482882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:56.483381 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:56.483617 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:56.483627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:56.484936 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:56.484941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:56.486185 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:56.486202 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:56.487206 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:56.487212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:56.487412 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:56.488250 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:56.488271 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:56.492625 (+ 4354us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":346,"threads_started":1}
W20260924 16:06:56.493152 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:56.481996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:45519 (local address 127.3.214.254:44483)
0924 16:06:56.482238 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:56.482243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:56.482475 (+ 232us) server_negotiation.cc:407] Connection header received
0924 16:06:56.482642 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:56.482646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:56.482726 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:56.482845 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:56.483777 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:56.484774 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:56.486326 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:56.487008 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:56.487171 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:56.492871 (+ 5700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.195:45519: BlockingRecv error: recv got EOF from 127.3.214.195:45519 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 24 16:06:56 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:56 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:56.830183 13404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:56.819032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.194:36377)
0924 16:06:56.819831 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:56.819851 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:56.820042 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:56.820436 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:56.820441 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:56.820969 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:56.821179 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:56.821189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:56.822429 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:56.822434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:56.823464 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:56.823478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:56.824370 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:56.824377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:56.824566 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:56.825318 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:56.825341 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:56.829814 (+ 4473us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":462,"threads_started":1}
W20260924 16:06:56.830489 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:56.819271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:36377 (local address 127.3.214.254:44483)
0924 16:06:56.819607 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:56.819615 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:56.819962 (+ 347us) server_negotiation.cc:407] Connection header received
0924 16:06:56.820199 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:56.820203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:56.820279 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:56.820375 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:56.821345 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:56.822271 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:56.823635 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:56.824133 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:56.824256 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:56.830165 (+ 5909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.194:36377: BlockingRecv error: recv got EOF from 127.3.214.194:36377 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 24 16:06:57 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:57 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:57.317297 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:57.306849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.193:38633)
0924 16:06:57.307432 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:57.307451 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:57.307625 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:57.308019 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:57.308024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:57.308580 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:57.308777 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:57.308788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:57.309973 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:57.309978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:57.311032 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:57.311049 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:57.311876 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:57.311881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:57.312026 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:57.312781 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:57.312803 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:57.317027 (+ 4224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":305,"threads_started":1}
W20260924 16:06:57.317656 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:57.306956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:38633 (local address 127.3.214.254:44483)
0924 16:06:57.307224 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:57.307231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:57.307552 (+ 321us) server_negotiation.cc:407] Connection header received
0924 16:06:57.307759 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:57.307765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:57.307837 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:57.307931 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:57.308939 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:57.309780 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:57.311210 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:57.311710 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:57.311814 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:57.317321 (+ 5507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.193:38633: BlockingRecv error: recv got EOF from 127.3.214.193:38633 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 16:06:57 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:57 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:57.504410 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:57.494075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.195:35529)
0924 16:06:57.494819 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:57.494837 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:57.494990 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:57.495376 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:57.495381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:57.495813 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:57.495990 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:57.495999 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:57.497071 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:57.497076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:57.498025 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:57.498038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:57.498828 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:57.498836 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:57.499003 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:57.499738 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:57.499756 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:57.504010 (+ 4254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":371,"threads_started":1}
W20260924 16:06:57.504558 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:57.494224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:35529 (local address 127.3.214.254:44483)
0924 16:06:57.494481 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:57.494488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:57.494951 (+ 463us) server_negotiation.cc:407] Connection header received
0924 16:06:57.495118 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:57.495123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:57.495199 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:57.495285 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:57.496143 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:57.496924 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:57.498181 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:57.498640 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:57.498750 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:57.504321 (+ 5571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.195:35529: BlockingRecv error: recv got EOF from 127.3.214.195:35529 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 16:06:57 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:57 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:57.843326 13407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:57.831795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.194:52221)
0924 16:06:57.832460 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:57.832481 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:57.832649 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:57.833185 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:57.833189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:57.833732 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:57.833935 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:57.833945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:57.835203 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:57.835208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:57.836317 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:57.836337 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:57.837361 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:57.837367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:57.837592 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:57.838427 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:57.838447 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:57.843015 (+ 4568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":374,"threads_started":1}
W20260924 16:06:57.843629 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:57.832033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:52221 (local address 127.3.214.254:44483)
0924 16:06:57.832220 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:57.832226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:57.832617 (+ 391us) server_negotiation.cc:407] Connection header received
0924 16:06:57.832825 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:57.832830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:57.832944 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:57.833069 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:57.834187 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:57.835001 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:57.836538 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:57.837182 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:57.837332 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:57.843354 (+ 6022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.194:52221: BlockingRecv error: recv got EOF from 127.3.214.194:52221 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 16:06:58 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:58 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.193@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:58.329599 13408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:58.318926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.193:40875)
0924 16:06:58.319643 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:58.319666 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:58.319833 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:58.320170 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:58.320175 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:58.320619 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:58.320830 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:58.320840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:58.321926 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:58.321933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:58.322926 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:58.322940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:58.323693 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:58.323700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:58.323895 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:58.324765 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:58.324786 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:58.329194 (+ 4408us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":393,"threads_started":1}
W20260924 16:06:58.329698 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:58.319125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:40875 (local address 127.3.214.254:44483)
0924 16:06:58.319437 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:58.319444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:58.319765 (+ 321us) server_negotiation.cc:407] Connection header received
0924 16:06:58.319967 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:58.319970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:58.320035 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:58.320119 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:58.320994 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:58.321760 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:58.323088 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:58.323528 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:58.323650 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:58.329443 (+ 5793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.193:40875: BlockingRecv error: recv got EOF from 127.3.214.193:40875 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Sep 24 16:06:58 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:58 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.195@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:58.516568 13409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:58.505677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.195:42281)
0924 16:06:58.506244 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:58.506278 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:58.506398 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:58.506785 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:58.506789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:58.507217 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:58.507418 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:58.507427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:58.508799 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:58.508803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:58.510059 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:58.510074 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:58.511049 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:58.511055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:58.511226 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:58.511921 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:58.511938 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:58.516162 (+ 4224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":292,"threads_started":1}
W20260924 16:06:58.516729 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:58.505963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:42281 (local address 127.3.214.254:44483)
0924 16:06:58.506306 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:58.506312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:58.506341 (+ 29us) server_negotiation.cc:407] Connection header received
0924 16:06:58.506523 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:58.506527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:58.506614 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:58.506738 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:58.507577 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:58.508614 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:58.510252 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:58.510855 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:58.511000 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:58.516454 (+ 5454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.195:42281: BlockingRecv error: recv got EOF from 127.3.214.195:42281 (error 108)
Metrics: {"server-negotiator.queue_time_us":183}
Sep 24 16:06:58 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:06:58 dist-test-slave-1gc3 krb5kdc[12354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.214.194@KRBTEST.COM for oryx/127.3.214.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:06:58.854775 13410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:58.844978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:44483 (local address 127.3.214.194:57125)
0924 16:06:58.845527 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0924 16:06:58.845545 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:06:58.845695 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:06:58.846133 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:06:58.846139 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:06:58.846601 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:06:58.846832 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:58.846842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:58.847970 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:58.847974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:58.848944 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:06:58.848955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:06:58.849729 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:06:58.849733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:06:58.849877 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:58.850565 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:06:58.850585 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:06:58.854472 (+ 3887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.214.254:44483: Server oryx/127.3.214.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":264,"threads_started":1}
W20260924 16:06:58.855262 13367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:06:58.845038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:57125 (local address 127.3.214.254:44483)
0924 16:06:58.845289 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:58.845297 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:58.845637 (+ 340us) server_negotiation.cc:407] Connection header received
0924 16:06:58.845840 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:58.845845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:58.845955 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:58.846050 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:58.846994 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:58.847823 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:58.849092 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:58.849581 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:58.849679 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:58.854885 (+ 5206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.214.194:57125: BlockingRecv error: recv got EOF from 127.3.214.194:57125 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 16:06:59.271382 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 12947
I20260924 16:06:59.293817 13412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:59.272945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40082 (local address 127.3.214.193:34195)
0924 16:06:59.273681 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:59.273690 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:59.273725 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:06:59.273809 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:59.273817 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:59.274182 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:59.274396 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:59.275962 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:59.277309 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:59.278742 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:59.279466 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:59.279583 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:59.289710 (+ 10127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:59.289725 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:59.289751 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:59.289826 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:59.293086 (+ 3260us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.214.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 16:06:59.293517 (+ 431us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40082: Request ticket server kudu/127.3.214.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":372,"threads_started":1}
W20260924 16:06:59.293957 13412 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40082: Request ticket server kudu/127.3.214.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 16:06:59.294298 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 12947: Not authorized: Client connection negotiation failed: client connection to 127.3.214.193:34195: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.214.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 16:06:59.294539 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 12947
I20260924 16:06:59.314687 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 13100
I20260924 16:06:59.330679 13413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:59.316105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41138 (local address 127.3.214.194:42951)
0924 16:06:59.317005 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:59.317018 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:59.317068 (+ 50us) server_negotiation.cc:407] Connection header received
0924 16:06:59.317171 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:59.317182 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:59.317594 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:59.317886 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:59.319277 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:59.320867 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:59.322346 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:59.323084 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:59.323210 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:59.326393 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:59.326405 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:59.326427 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:59.326487 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:59.329717 (+ 3230us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.214.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 16:06:59.330335 (+ 618us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41138: Request ticket server kudu/127.3.214.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":414,"threads_started":1}
W20260924 16:06:59.330814 13413 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41138: Request ticket server kudu/127.3.214.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 16:06:59.330914 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 13100: Not authorized: Client connection negotiation failed: client connection to 127.3.214.194:42951: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.214.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 16:06:59.331312 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 13100
I20260924 16:06:59.350688 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 13234
I20260924 16:06:59.369586 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:06:59.352454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53792 (local address 127.3.214.195:34035)
0924 16:06:59.354748 (+ 2294us) server_negotiation.cc:206] Beginning negotiation
0924 16:06:59.354757 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:06:59.354793 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:06:59.354879 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:06:59.354888 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:06:59.355206 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:06:59.355434 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:06:59.357382 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:59.359314 (+ 1932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:59.360863 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:06:59.361486 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:06:59.361617 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:06:59.365375 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:06:59.365387 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:06:59.365410 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:06:59.365464 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:06:59.368855 (+ 3391us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.214.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 16:06:59.369278 (+ 423us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53792: Request ticket server kudu/127.3.214.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":1984,"thread_start_us":1849,"threads_started":1}
W20260924 16:06:59.369726 13414 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53792: Request ticket server kudu/127.3.214.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 16:06:59.369953 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 13234: Not authorized: Client connection negotiation failed: client connection to 127.3.214.195:34035: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.214.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 16:06:59.370183 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 13234
I20260924 16:06:59.389735 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 12877
I20260924 16:06:59.391661 12940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:06:59.533898 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 12877
2026-09-24T16:06:59Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22439 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T16:06:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:06:59Z Disabled control of system clock
I20260924 16:06:59.625761 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:39765
--webserver_interface=127.3.214.254
--webserver_port=0
--builtin_ntp_servers=127.3.214.212:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:39765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 16:07:00.041272 13419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:07:00.041710 13419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:07:00.041806 13419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:07:00.050161 13419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:07:00.050314 13419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:07:00.050354 13419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:07:00.050420 13419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:07:00.050470 13419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:07:00.061419 13419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:39765
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:39765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:07:00.063488 13419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:07:00.065984 13419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:07:00.080565 13424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:07:00.080474 13427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:07:00.080690 13419 server_base.cc:1034] running on GCE node
W20260924 16:07:00.080487 13425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:07:00.082279 13419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:07:00.084196 13419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:07:00.085460 13419 hybrid_clock.cc:648] HybridClock initialized: now 1790266020085388 us; error 47 us; skew 500 ppm
I20260924 16:07:00.085930 13419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:07:00.089160 13419 webserver.cc:460] Webserver started at http://127.3.214.254:46317/ using document root <none> and password file <none>
I20260924 16:07:00.090201 13419 fs_manager.cc:359] Metadata directory not provided
I20260924 16:07:00.090349 13419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:07:00.090785 13419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:07:00.094713 13419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "ddddb890eee34a228d73878afc2fad20"
format_stamp: "Formatted at 2026-09-24 16:07:00 on dist-test-slave-1gc3"
server_key: "67345ff68868490744846eda24106f80"
server_key_iv: "37bfef7843bf0c5b5f6075715e6b7c18"
server_key_version: "encryptionkey@0"
I20260924 16:07:00.095777 13419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "ddddb890eee34a228d73878afc2fad20"
format_stamp: "Formatted at 2026-09-24 16:07:00 on dist-test-slave-1gc3"
server_key: "67345ff68868490744846eda24106f80"
server_key_iv: "37bfef7843bf0c5b5f6075715e6b7c18"
server_key_version: "encryptionkey@0"
I20260924 16:07:00.102991 13419 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260924 16:07:00.107882 13434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:00.109776 13419 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:07:00.110016 13419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "ddddb890eee34a228d73878afc2fad20"
format_stamp: "Formatted at 2026-09-24 16:07:00 on dist-test-slave-1gc3"
server_key: "67345ff68868490744846eda24106f80"
server_key_iv: "37bfef7843bf0c5b5f6075715e6b7c18"
server_key_version: "encryptionkey@0"
I20260924 16:07:00.110314 13419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:07:00.130856 13419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:07:00.132085 13419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:07:00.132416 13419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:07:00.155879 13485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:39765 every 8 connection(s)
I20260924 16:07:00.155871 13419 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:39765
I20260924 16:07:00.157871 13419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:07:00.158931 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 13419
I20260924 16:07:00.159186 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:07:00.160238 3931 external_mini_cluster.cc:1428] Setting key 4d1e75dca242632d6eae44f00e3a45aa
I20260924 16:07:00.163722 13486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:07:00.180546 13486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddddb890eee34a228d73878afc2fad20: Bootstrap starting.
I20260924 16:07:00.181759 13489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:00.163575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57520 (local address 127.3.214.254:39765)
0924 16:07:00.167293 (+ 3718us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:00.167311 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:00.168126 (+ 815us) server_negotiation.cc:407] Connection header received
0924 16:07:00.170431 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:00.170467 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:00.171267 (+ 800us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:00.171811 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:00.172617 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:00.174191 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:00.175823 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:00.176710 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:00.176948 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:00.178588 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:07:00.178598 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:07:00.178623 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:07:00.179381 (+ 758us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:07:00.179492 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:07:00.179526 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:07:00.179659 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:00.179945 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 16:07:00.180397 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3033,"thread_start_us":2680,"threads_started":1}
I20260924 16:07:00.184727 13486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddddb890eee34a228d73878afc2fad20: Neither blocks nor log segments found. Creating new log.
I20260924 16:07:00.186122 13486 log.cc:824] T 00000000000000000000000000000000 P ddddb890eee34a228d73878afc2fad20: Log is configured to *not* fsync() on all Append() calls
I20260924 16:07:00.189975 13486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddddb890eee34a228d73878afc2fad20: No bootstrap required, opened a new log
I20260924 16:07:00.198140 13486 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddddb890eee34a228d73878afc2fad20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddddb890eee34a228d73878afc2fad20" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39765 } }
I20260924 16:07:00.198619 13486 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddddb890eee34a228d73878afc2fad20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:07:00.198715 13486 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddddb890eee34a228d73878afc2fad20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddddb890eee34a228d73878afc2fad20, State: Initialized, Role: FOLLOWER
I20260924 16:07:00.199353 13486 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddddb890eee34a228d73878afc2fad20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddddb890eee34a228d73878afc2fad20" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39765 } }
I20260924 16:07:00.199604 13486 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddddb890eee34a228d73878afc2fad20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:07:00.199735 13486 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddddb890eee34a228d73878afc2fad20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:07:00.199908 13486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddddb890eee34a228d73878afc2fad20 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:07:00.202215 13486 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddddb890eee34a228d73878afc2fad20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddddb890eee34a228d73878afc2fad20" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39765 } }
I20260924 16:07:00.202796 13486 leader_election.cc:302] T 00000000000000000000000000000000 P ddddb890eee34a228d73878afc2fad20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddddb890eee34a228d73878afc2fad20; no voters:
I20260924 16:07:00.203514 13486 leader_election.cc:288] T 00000000000000000000000000000000 P ddddb890eee34a228d73878afc2fad20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:07:00.203830 13491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddddb890eee34a228d73878afc2fad20 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:07:00.204723 13491 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddddb890eee34a228d73878afc2fad20 [term 1 LEADER]: Becoming Leader. State: Replica: ddddb890eee34a228d73878afc2fad20, State: Running, Role: LEADER
I20260924 16:07:00.205574 13491 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddddb890eee34a228d73878afc2fad20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddddb890eee34a228d73878afc2fad20" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39765 } }
I20260924 16:07:00.206055 13486 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddddb890eee34a228d73878afc2fad20 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:07:00.209482 13493 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddddb890eee34a228d73878afc2fad20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddddb890eee34a228d73878afc2fad20. Latest consensus state: current_term: 1 leader_uuid: "ddddb890eee34a228d73878afc2fad20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddddb890eee34a228d73878afc2fad20" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39765 } } }
I20260924 16:07:00.209586 13492 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddddb890eee34a228d73878afc2fad20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddddb890eee34a228d73878afc2fad20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddddb890eee34a228d73878afc2fad20" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 39765 } } }
I20260924 16:07:00.210196 13493 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddddb890eee34a228d73878afc2fad20 [sys.catalog]: This master's current role is: LEADER
I20260924 16:07:00.210290 13492 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddddb890eee34a228d73878afc2fad20 [sys.catalog]: This master's current role is: LEADER
I20260924 16:07:00.212136 13499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:07:00.219313 13499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:07:00.231125 13499 catalog_manager.cc:1347] Generated new cluster ID: 1328a32a919f4ebb9e236fa35e649217
I20260924 16:07:00.231316 13499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:07:00.245759 13499 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:07:00.247304 13499 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:07:00.259366 13499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddddb890eee34a228d73878afc2fad20: Generated new TSK 0
I20260924 16:07:00.260370 13499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:07:00.270424 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39765
--builtin_ntp_servers=127.3.214.212:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 16:07:00.664600 13510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:07:00.664959 13510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:07:00.665097 13510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:07:00.672770 13510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:07:00.672907 13510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:07:00.673067 13510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:07:00.683314 13510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:07:00.684826 13510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:07:00.686910 13510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:07:00.702272 13516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:07:00.702320 13515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:07:00.702376 13518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:07:00.703440 13510 server_base.cc:1034] running on GCE node
I20260924 16:07:00.704173 13510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:07:00.705314 13510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:07:00.706543 13510 hybrid_clock.cc:648] HybridClock initialized: now 1790266020706465 us; error 68 us; skew 500 ppm
I20260924 16:07:00.706990 13510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:07:00.709870 13510 webserver.cc:460] Webserver started at http://127.3.214.193:41711/ using document root <none> and password file <none>
I20260924 16:07:00.710728 13510 fs_manager.cc:359] Metadata directory not provided
I20260924 16:07:00.710844 13510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:07:00.711187 13510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:07:00.714499 13510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "497923557a5b45e19e8ea8771fa79a81"
format_stamp: "Formatted at 2026-09-24 16:07:00 on dist-test-slave-1gc3"
server_key: "fb110ce6b39b652f7a43f31ac572c864"
server_key_iv: "63d3ba9f9e374faa4f88ed4f5547d9f4"
server_key_version: "encryptionkey@0"
I20260924 16:07:00.715413 13510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "497923557a5b45e19e8ea8771fa79a81"
format_stamp: "Formatted at 2026-09-24 16:07:00 on dist-test-slave-1gc3"
server_key: "fb110ce6b39b652f7a43f31ac572c864"
server_key_iv: "63d3ba9f9e374faa4f88ed4f5547d9f4"
server_key_version: "encryptionkey@0"
I20260924 16:07:00.721716 13510 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 16:07:00.726372 13524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:00.728332 13510 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:07:00.728583 13510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "497923557a5b45e19e8ea8771fa79a81"
format_stamp: "Formatted at 2026-09-24 16:07:00 on dist-test-slave-1gc3"
server_key: "fb110ce6b39b652f7a43f31ac572c864"
server_key_iv: "63d3ba9f9e374faa4f88ed4f5547d9f4"
server_key_version: "encryptionkey@0"
I20260924 16:07:00.728845 13510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:07:00.750038 13510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:07:00.751240 13510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:07:00.751569 13510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:07:00.752828 13510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:07:00.754916 13510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:07:00.755081 13510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260924 16:07:00.755221 13510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:07:00.755319 13510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:00.795799 13510 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:46499
I20260924 16:07:00.795817 13637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:46499 every 8 connection(s)
I20260924 16:07:00.797515 13510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:07:00.806211 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 13510
I20260924 16:07:00.806422 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:07:00.807163 3931 external_mini_cluster.cc:1428] Setting key d13b26cc99b14f055069d930ef58e24e
I20260924 16:07:00.810854 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39765
--builtin_ntp_servers=127.3.214.212:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 16:07:00.814795 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:00.801223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:44483 (local address 127.3.214.254:39765)
0924 16:07:00.801746 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:00.801751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:00.803207 (+ 1456us) server_negotiation.cc:407] Connection header received
0924 16:07:00.805322 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:00.805328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:00.805431 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:00.805541 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:00.807189 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:00.808082 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:00.810663 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:00.811304 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:00.811437 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:00.813662 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:07:00.813669 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:07:00.813672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:07:00.813699 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:07:00.813769 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:07:00.813789 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:07:00.813893 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:00.814306 (+ 413us) server_negotiation.cc:299] Negotiation successful
0924 16:07:00.814439 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":257,"threads_started":1}
I20260924 16:07:00.816606 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:00.801788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39765 (local address 127.3.214.193:44483)
0924 16:07:00.802992 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0924 16:07:00.803046 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 16:07:00.804985 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:07:00.805862 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:07:00.805895 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:07:00.806021 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:07:00.806962 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:07:00.806988 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:07:00.808438 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:07:00.808446 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:07:00.810429 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:07:00.810458 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:07:00.811573 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:07:00.811580 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:07:00.811837 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:00.813021 (+ 1184us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:07:00.813025 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:07:00.813118 (+ 93us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:07:00.813305 (+ 187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:07:00.813915 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:07:00.813926 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:07:00.813937 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 16:07:00.814214 (+ 277us) client_negotiation.cc:240] Negotiation successful
0924 16:07:00.814846 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":378,"threads_started":1}
I20260924 16:07:00.818630 13638 heartbeater.cc:347] Connected to a master server at 127.3.214.254:39765
I20260924 16:07:00.819093 13638 heartbeater.cc:464] Registering TS with master...
I20260924 16:07:00.820339 13638 heartbeater.cc:511] Master 127.3.214.254:39765 requested a full tablet report, sending...
I20260924 16:07:00.823305 13451 ts_manager.cc:194] Registered new tserver with Master: 497923557a5b45e19e8ea8771fa79a81 (127.3.214.193:46499)
I20260924 16:07:00.825438 13451 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.214.193:44483
W20260924 16:07:01.193334 13642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:07:01.193770 13642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:07:01.193878 13642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:07:01.202045 13642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:07:01.202240 13642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:07:01.202375 13642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:07:01.212965 13642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:07:01.214905 13642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:07:01.217389 13642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:07:01.233700 13650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:07:01.233795 13648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:07:01.233639 13647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:07:01.235204 13642 server_base.cc:1034] running on GCE node
I20260924 16:07:01.236055 13642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:07:01.237402 13642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:07:01.238632 13642 hybrid_clock.cc:648] HybridClock initialized: now 1790266021238564 us; error 61 us; skew 500 ppm
I20260924 16:07:01.239037 13642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:07:01.242090 13642 webserver.cc:460] Webserver started at http://127.3.214.194:45123/ using document root <none> and password file <none>
I20260924 16:07:01.243129 13642 fs_manager.cc:359] Metadata directory not provided
I20260924 16:07:01.243249 13642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:07:01.243639 13642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:07:01.247440 13642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "671d9281468848d19ee137b106219274"
format_stamp: "Formatted at 2026-09-24 16:07:01 on dist-test-slave-1gc3"
server_key: "ab19edd8585cfdb379c01c9001acc6a2"
server_key_iv: "b0e124f8b19a995ffb9d6af7d2c69e5d"
server_key_version: "encryptionkey@0"
I20260924 16:07:01.248404 13642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "671d9281468848d19ee137b106219274"
format_stamp: "Formatted at 2026-09-24 16:07:01 on dist-test-slave-1gc3"
server_key: "ab19edd8585cfdb379c01c9001acc6a2"
server_key_iv: "b0e124f8b19a995ffb9d6af7d2c69e5d"
server_key_version: "encryptionkey@0"
I20260924 16:07:01.255330 13642 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 16:07:01.259928 13656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:01.261718 13642 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:07:01.261943 13642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "671d9281468848d19ee137b106219274"
format_stamp: "Formatted at 2026-09-24 16:07:01 on dist-test-slave-1gc3"
server_key: "ab19edd8585cfdb379c01c9001acc6a2"
server_key_iv: "b0e124f8b19a995ffb9d6af7d2c69e5d"
server_key_version: "encryptionkey@0"
I20260924 16:07:01.262243 13642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:07:01.280318 13642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:07:01.281430 13642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:07:01.281764 13642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:07:01.283006 13642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:07:01.284890 13642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:07:01.284999 13642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:01.285125 13642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:07:01.285197 13642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:01.322634 13642 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:42279
I20260924 16:07:01.322702 13769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:42279 every 8 connection(s)
I20260924 16:07:01.324321 13642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:07:01.326777 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 13642
I20260924 16:07:01.326983 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:07:01.327612 3931 external_mini_cluster.cc:1428] Setting key 8133c7f27276d79953ea36ba2b86ec88
I20260924 16:07:01.331128 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39765
--builtin_ntp_servers=127.3.214.212:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 16:07:01.341795 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:01.327865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:54433 (local address 127.3.214.254:39765)
0924 16:07:01.328553 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:01.328561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:01.330218 (+ 1657us) server_negotiation.cc:407] Connection header received
0924 16:07:01.332524 (+ 2306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:01.332530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:01.332633 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:01.332807 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:01.334767 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:01.335780 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:01.337885 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:01.338522 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:01.338661 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:01.340846 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:07:01.340851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:07:01.340855 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:07:01.340879 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:07:01.340932 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:07:01.340946 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:07:01.341038 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:01.341390 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 16:07:01.341492 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":271,"threads_started":1}
I20260924 16:07:01.343591 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:01.328581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39765 (local address 127.3.214.194:54433)
0924 16:07:01.329981 (+ 1400us) negotiation.cc:107] Waiting for socket to connect
0924 16:07:01.330037 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 16:07:01.332136 (+ 2099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:07:01.333279 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:07:01.333292 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:07:01.333441 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:07:01.334487 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:07:01.334528 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:07:01.336034 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:07:01.336040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:07:01.337649 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:07:01.337669 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:07:01.338896 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:07:01.338904 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:07:01.339298 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:01.340331 (+ 1033us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:07:01.340334 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:07:01.340405 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:07:01.340548 (+ 143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:07:01.341060 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:07:01.341070 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:07:01.341079 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 16:07:01.341304 (+ 225us) client_negotiation.cc:240] Negotiation successful
0924 16:07:01.341884 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":418,"threads_started":1}
I20260924 16:07:01.345700 13770 heartbeater.cc:347] Connected to a master server at 127.3.214.254:39765
I20260924 16:07:01.346194 13770 heartbeater.cc:464] Registering TS with master...
I20260924 16:07:01.347209 13770 heartbeater.cc:511] Master 127.3.214.254:39765 requested a full tablet report, sending...
I20260924 16:07:01.349233 13451 ts_manager.cc:194] Registered new tserver with Master: 671d9281468848d19ee137b106219274 (127.3.214.194:42279)
I20260924 16:07:01.350457 13451 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.214.194:54433
W20260924 16:07:01.748293 13774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:07:01.748721 13774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:07:01.748824 13774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:07:01.757560 13774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:07:01.757721 13774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:07:01.757849 13774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:07:01.768486 13774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:39765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:07:01.770882 13774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:07:01.773663 13774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:07:01.790490 13779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:07:01.790622 13780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:07:01.790611 13783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:07:01.791674 13774 server_base.cc:1034] running on GCE node
I20260924 16:07:01.792531 13774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:07:01.793932 13774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:07:01.795446 13774 hybrid_clock.cc:648] HybridClock initialized: now 1790266021795336 us; error 83 us; skew 500 ppm
I20260924 16:07:01.795926 13774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:07:01.800585 13774 webserver.cc:460] Webserver started at http://127.3.214.195:36081/ using document root <none> and password file <none>
I20260924 16:07:01.801524 13774 fs_manager.cc:359] Metadata directory not provided
I20260924 16:07:01.801637 13774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:07:01.802024 13774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:07:01.806730 13774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "830fc1df0df7404daf8aef777b34eb2f"
format_stamp: "Formatted at 2026-09-24 16:07:01 on dist-test-slave-1gc3"
server_key: "e4372df928a502301ff104d2445f0142"
server_key_iv: "d1f94b06b7fdc53a393c196c728d1075"
server_key_version: "encryptionkey@0"
I20260924 16:07:01.807818 13774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "830fc1df0df7404daf8aef777b34eb2f"
format_stamp: "Formatted at 2026-09-24 16:07:01 on dist-test-slave-1gc3"
server_key: "e4372df928a502301ff104d2445f0142"
server_key_iv: "d1f94b06b7fdc53a393c196c728d1075"
server_key_version: "encryptionkey@0"
I20260924 16:07:01.814918 13774 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260924 16:07:01.820119 13789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:01.822487 13774 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:07:01.822743 13774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "830fc1df0df7404daf8aef777b34eb2f"
format_stamp: "Formatted at 2026-09-24 16:07:01 on dist-test-slave-1gc3"
server_key: "e4372df928a502301ff104d2445f0142"
server_key_iv: "d1f94b06b7fdc53a393c196c728d1075"
server_key_version: "encryptionkey@0"
I20260924 16:07:01.823017 13774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:07:01.829941 13638 heartbeater.cc:503] Master 127.3.214.254:39765 was elected leader, sending a full tablet report...
I20260924 16:07:01.841753 13774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:07:01.842835 13774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:07:01.843133 13774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:07:01.844276 13774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:07:01.846014 13774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:07:01.846155 13774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:01.846284 13774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:07:01.846350 13774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:01.883371 13774 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:33991
I20260924 16:07:01.883486 13902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:33991 every 8 connection(s)
I20260924 16:07:01.884883 13774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:07:01.886761 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 13774
I20260924 16:07:01.886955 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:07:01.887435 3931 external_mini_cluster.cc:1428] Setting key ce1d07d3028f281a35db2ef86e752b68
I20260924 16:07:01.906914 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:01.888559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:36327 (local address 127.3.214.254:39765)
0924 16:07:01.897940 (+ 9381us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:01.897948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:01.897984 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:07:01.898304 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:01.898309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:01.898426 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:01.898542 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:01.900075 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:01.900901 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:01.903255 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:01.903833 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:01.903973 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:01.905881 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:07:01.905887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:07:01.905891 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:07:01.905916 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:07:01.905984 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:07:01.906002 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:07:01.906120 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:01.906477 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 16:07:01.906588 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9181,"thread_start_us":8892,"threads_started":1}
I20260924 16:07:01.908355 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:01.889359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:39765 (local address 127.3.214.195:36327)
0924 16:07:01.895342 (+ 5983us) negotiation.cc:107] Waiting for socket to connect
0924 16:07:01.895417 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0924 16:07:01.897822 (+ 2405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:07:01.898895 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:07:01.898906 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:07:01.899034 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:07:01.899878 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:07:01.899900 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:07:01.901077 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:07:01.901081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:07:01.903031 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:07:01.903052 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:07:01.904127 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:07:01.904134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:07:01.904414 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:01.905365 (+ 951us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:07:01.905370 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:07:01.905431 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:07:01.905567 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:07:01.906177 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:07:01.906191 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:07:01.906202 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 16:07:01.906420 (+ 218us) client_negotiation.cc:240] Negotiation successful
0924 16:07:01.906876 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5182,"thread_start_us":4884,"threads_started":1}
I20260924 16:07:01.910180 13903 heartbeater.cc:347] Connected to a master server at 127.3.214.254:39765
I20260924 16:07:01.910553 13903 heartbeater.cc:464] Registering TS with master...
I20260924 16:07:01.911577 13903 heartbeater.cc:511] Master 127.3.214.254:39765 requested a full tablet report, sending...
I20260924 16:07:01.913579 13451 ts_manager.cc:194] Registered new tserver with Master: 830fc1df0df7404daf8aef777b34eb2f (127.3.214.195:33991)
I20260924 16:07:01.914635 13451 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.214.195:36327
I20260924 16:07:01.917955 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 16:07:01.926064 13905 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:07:01.923931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57522 (local address 127.3.214.254:39765)
0924 16:07:01.924293 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:01.924302 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:01.924710 (+ 408us) server_negotiation.cc:407] Connection header received
0924 16:07:01.924960 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:01.924965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:01.925060 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:01.925160 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:01.925746 (+ 586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57522: BlockingRecv error: recv got EOF from 127.0.0.1:57522 (error 108)
Metrics: {"server-negotiator.queue_time_us":152}
I20260924 16:07:01.926800 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 13510
I20260924 16:07:01.936985 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:01.928536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53674 (local address 127.3.214.193:46499)
0924 16:07:01.929276 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:01.929284 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:01.929314 (+ 30us) server_negotiation.cc:407] Connection header received
0924 16:07:01.929389 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:01.929396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:01.929638 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:01.929790 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:01.930523 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:01.931611 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:01.933173 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:01.933838 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:01.933988 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:01.935381 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:07:01.935392 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:07:01.935411 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:07:01.936082 (+ 671us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:07:01.936139 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:07:01.936161 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:07:01.936278 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:01.936465 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 16:07:01.936621 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":303,"threads_started":1}
I20260924 16:07:01.938664 13633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:07:02.028945 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 13510
I20260924 16:07:02.047802 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 13642
I20260924 16:07:02.057328 13915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:02.048962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52936 (local address 127.3.214.194:42279)
0924 16:07:02.049626 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:02.049639 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:02.049679 (+ 40us) server_negotiation.cc:407] Connection header received
0924 16:07:02.049770 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:02.049778 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:02.050079 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:02.050290 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:02.050738 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:02.051892 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:02.053140 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:02.053873 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:02.054002 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:02.055287 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:07:02.055298 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:07:02.055319 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:07:02.056141 (+ 822us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:07:02.056221 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:07:02.056246 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:07:02.056363 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:02.056639 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 16:07:02.056866 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":353,"threads_started":1}
I20260924 16:07:02.059489 13765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:07:02.151075 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 13642
I20260924 16:07:02.175206 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 13774
I20260924 16:07:02.185508 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:02.176436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56330 (local address 127.3.214.195:33991)
0924 16:07:02.177212 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:02.177221 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:02.177262 (+ 41us) server_negotiation.cc:407] Connection header received
0924 16:07:02.177354 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:02.177363 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:02.177685 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:02.177857 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:02.178553 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:02.179832 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:02.181246 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:02.181984 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:02.182259 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:02.183602 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:07:02.183615 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:07:02.183639 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:07:02.184469 (+ 830us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:07:02.184533 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:07:02.184560 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:07:02.184717 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:02.184904 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 16:07:02.185065 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":354,"threads_started":1}
I20260924 16:07:02.187441 13898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:07:02.279759 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 13774
I20260924 16:07:02.296859 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 13419
I20260924 16:07:02.298409 13481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:07:02.420754 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 13419
2026-09-24T16:07:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2879 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T16:07:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:07:02Z Disabled control of system clock
I20260924 16:07:02.505663 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:33899
--webserver_interface=127.3.214.254
--webserver_port=0
--builtin_ntp_servers=127.3.214.212:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:33899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 16:07:02.922032 13923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:07:02.922499 13923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:07:02.922648 13923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:07:02.931267 13923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:07:02.931435 13923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:07:02.931475 13923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:07:02.931576 13923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:07:02.931644 13923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:07:02.942430 13923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:33899
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:33899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:07:02.944242 13923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:07:02.946620 13923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:07:02.960736 13928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:07:02.960831 13929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:07:02.960809 13931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:07:02.961804 13923 server_base.cc:1034] running on GCE node
I20260924 16:07:02.962913 13923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:07:02.964829 13923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:07:02.966063 13923 hybrid_clock.cc:648] HybridClock initialized: now 1790266022966006 us; error 47 us; skew 500 ppm
I20260924 16:07:02.966547 13923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:07:02.969415 13923 webserver.cc:460] Webserver started at http://127.3.214.254:45191/ using document root <none> and password file <none>
I20260924 16:07:02.970371 13923 fs_manager.cc:359] Metadata directory not provided
I20260924 16:07:02.970496 13923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:07:02.970880 13923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:07:02.974548 13923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "2fc96a34ac98428caf203d3ed05cf031"
format_stamp: "Formatted at 2026-09-24 16:07:02 on dist-test-slave-1gc3"
server_key: "ae24492075f70fb84236810e9162203c"
server_key_iv: "81dfcfe5b0f3b4e3112b6dc179ee369e"
server_key_version: "encryptionkey@0"
I20260924 16:07:02.975492 13923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "2fc96a34ac98428caf203d3ed05cf031"
format_stamp: "Formatted at 2026-09-24 16:07:02 on dist-test-slave-1gc3"
server_key: "ae24492075f70fb84236810e9162203c"
server_key_iv: "81dfcfe5b0f3b4e3112b6dc179ee369e"
server_key_version: "encryptionkey@0"
I20260924 16:07:02.982255 13923 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 16:07:02.986865 13937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:02.988883 13923 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:07:02.989101 13923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "2fc96a34ac98428caf203d3ed05cf031"
format_stamp: "Formatted at 2026-09-24 16:07:02 on dist-test-slave-1gc3"
server_key: "ae24492075f70fb84236810e9162203c"
server_key_iv: "81dfcfe5b0f3b4e3112b6dc179ee369e"
server_key_version: "encryptionkey@0"
I20260924 16:07:02.989364 13923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:07:02.996341 13923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:07:02.997354 13923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:07:02.997696 13923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:07:03.020283 13923 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:33899
I20260924 16:07:03.020341 13989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:33899 every 8 connection(s)
I20260924 16:07:03.022161 13923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:07:03.027005 13990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:07:03.029013 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 13923
I20260924 16:07:03.029278 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:07:03.030174 3931 external_mini_cluster.cc:1428] Setting key 840e630a5fdd2592681cab24bb480a16
I20260924 16:07:03.041997 13990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fc96a34ac98428caf203d3ed05cf031: Bootstrap starting.
I20260924 16:07:03.042006 13993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:03.033187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47490 (local address 127.3.214.254:33899)
0924 16:07:03.034751 (+ 1564us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:03.034767 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:03.034831 (+ 64us) server_negotiation.cc:407] Connection header received
0924 16:07:03.037054 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:03.037088 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:03.037893 (+ 805us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:03.038515 (+ 622us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:03.039833 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:07:03.039841 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:07:03.039856 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:07:03.039875 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:07:03.039956 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:07:03.039978 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:07:03.040213 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:03.040348 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 16:07:03.040785 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":955,"thread_start_us":579,"threads_started":1}
I20260924 16:07:03.046046 13990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fc96a34ac98428caf203d3ed05cf031: Neither blocks nor log segments found. Creating new log.
I20260924 16:07:03.047259 13990 log.cc:824] T 00000000000000000000000000000000 P 2fc96a34ac98428caf203d3ed05cf031: Log is configured to *not* fsync() on all Append() calls
I20260924 16:07:03.051318 13990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fc96a34ac98428caf203d3ed05cf031: No bootstrap required, opened a new log
I20260924 16:07:03.060384 13990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fc96a34ac98428caf203d3ed05cf031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc96a34ac98428caf203d3ed05cf031" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 33899 } }
I20260924 16:07:03.060844 13990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fc96a34ac98428caf203d3ed05cf031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:07:03.060943 13990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fc96a34ac98428caf203d3ed05cf031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc96a34ac98428caf203d3ed05cf031, State: Initialized, Role: FOLLOWER
I20260924 16:07:03.061583 13990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fc96a34ac98428caf203d3ed05cf031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc96a34ac98428caf203d3ed05cf031" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 33899 } }
I20260924 16:07:03.061836 13990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fc96a34ac98428caf203d3ed05cf031 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:07:03.061971 13990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fc96a34ac98428caf203d3ed05cf031 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:07:03.062196 13990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fc96a34ac98428caf203d3ed05cf031 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:07:03.064814 13990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fc96a34ac98428caf203d3ed05cf031 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc96a34ac98428caf203d3ed05cf031" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 33899 } }
I20260924 16:07:03.065500 13990 leader_election.cc:302] T 00000000000000000000000000000000 P 2fc96a34ac98428caf203d3ed05cf031 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fc96a34ac98428caf203d3ed05cf031; no voters:
I20260924 16:07:03.066234 13990 leader_election.cc:288] T 00000000000000000000000000000000 P 2fc96a34ac98428caf203d3ed05cf031 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:07:03.066483 13995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fc96a34ac98428caf203d3ed05cf031 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:07:03.067368 13995 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fc96a34ac98428caf203d3ed05cf031 [term 1 LEADER]: Becoming Leader. State: Replica: 2fc96a34ac98428caf203d3ed05cf031, State: Running, Role: LEADER
I20260924 16:07:03.068096 13995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fc96a34ac98428caf203d3ed05cf031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc96a34ac98428caf203d3ed05cf031" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 33899 } }
I20260924 16:07:03.068764 13990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fc96a34ac98428caf203d3ed05cf031 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:07:03.071324 13997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fc96a34ac98428caf203d3ed05cf031 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fc96a34ac98428caf203d3ed05cf031. Latest consensus state: current_term: 1 leader_uuid: "2fc96a34ac98428caf203d3ed05cf031" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc96a34ac98428caf203d3ed05cf031" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 33899 } } }
I20260924 16:07:03.071833 13997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fc96a34ac98428caf203d3ed05cf031 [sys.catalog]: This master's current role is: LEADER
I20260924 16:07:03.071744 13996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fc96a34ac98428caf203d3ed05cf031 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fc96a34ac98428caf203d3ed05cf031" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc96a34ac98428caf203d3ed05cf031" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 33899 } } }
I20260924 16:07:03.072130 13996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fc96a34ac98428caf203d3ed05cf031 [sys.catalog]: This master's current role is: LEADER
I20260924 16:07:03.073704 14000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:07:03.080508 14000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:07:03.093102 14000 catalog_manager.cc:1347] Generated new cluster ID: 940b5c73a63e416dbb1c19dd93e24875
I20260924 16:07:03.093235 14000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:07:03.119932 14000 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:07:03.120463 14000 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:07:03.130249 14000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fc96a34ac98428caf203d3ed05cf031: Generated new TSK 0
I20260924 16:07:03.131315 14000 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:07:03.149026 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:33899
--builtin_ntp_servers=127.3.214.212:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 16:07:03.532011 14014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:07:03.532337 14014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:07:03.532462 14014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:07:03.540512 14014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:07:03.540683 14014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:07:03.540863 14014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:07:03.551285 14014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:33899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:07:03.552656 14014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:07:03.554625 14014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:07:03.570652 14022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:07:03.570637 14019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:07:03.571302 14020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:07:03.576013 14014 server_base.cc:1034] running on GCE node
I20260924 16:07:03.576864 14014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:07:03.578059 14014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:07:03.579308 14014 hybrid_clock.cc:648] HybridClock initialized: now 1790266023579214 us; error 99 us; skew 500 ppm
I20260924 16:07:03.579679 14014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:07:03.582441 14014 webserver.cc:460] Webserver started at http://127.3.214.193:40119/ using document root <none> and password file <none>
I20260924 16:07:03.583190 14014 fs_manager.cc:359] Metadata directory not provided
I20260924 16:07:03.583298 14014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:07:03.583645 14014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:07:03.586879 14014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "96a8c3d0804043a095452ee4468154c5"
format_stamp: "Formatted at 2026-09-24 16:07:03 on dist-test-slave-1gc3"
server_key: "d8c2b78f1a550b50a59db77b181f5944"
server_key_iv: "f4696c00f65b7ff41dc4d1385f73c63d"
server_key_version: "encryptionkey@0"
I20260924 16:07:03.587783 14014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "96a8c3d0804043a095452ee4468154c5"
format_stamp: "Formatted at 2026-09-24 16:07:03 on dist-test-slave-1gc3"
server_key: "d8c2b78f1a550b50a59db77b181f5944"
server_key_iv: "f4696c00f65b7ff41dc4d1385f73c63d"
server_key_version: "encryptionkey@0"
I20260924 16:07:03.595372 14014 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260924 16:07:03.599895 14028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:03.601786 14014 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:07:03.602010 14014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "96a8c3d0804043a095452ee4468154c5"
format_stamp: "Formatted at 2026-09-24 16:07:03 on dist-test-slave-1gc3"
server_key: "d8c2b78f1a550b50a59db77b181f5944"
server_key_iv: "f4696c00f65b7ff41dc4d1385f73c63d"
server_key_version: "encryptionkey@0"
I20260924 16:07:03.602327 14014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:07:03.609102 14014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:07:03.610069 14014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:07:03.610409 14014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:07:03.611526 14014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:07:03.613164 14014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:07:03.613263 14014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:03.613389 14014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:07:03.613476 14014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:03.651266 14014 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:33077
I20260924 16:07:03.651365 14141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:33077 every 8 connection(s)
I20260924 16:07:03.652755 14014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:07:03.658571 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 14014
I20260924 16:07:03.658780 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:07:03.659379 3931 external_mini_cluster.cc:1428] Setting key f2e89da5307f217a8fb79d513235736e
I20260924 16:07:03.662616 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:33899
--builtin_ntp_servers=127.3.214.212:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 16:07:03.663678 14144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:03.656155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:55691 (local address 127.3.214.254:33899)
0924 16:07:03.656735 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:03.656741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:03.658348 (+ 1607us) server_negotiation.cc:407] Connection header received
0924 16:07:03.660608 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:03.660614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:03.660707 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:03.660818 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:03.662744 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:07:03.662751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:07:03.662755 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:07:03.662783 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:07:03.662846 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:07:03.662862 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:07:03.663013 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:03.663216 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 16:07:03.663303 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":269,"threads_started":1}
I20260924 16:07:03.665058 14145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:03.656786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:33899 (local address 127.3.214.193:55691)
0924 16:07:03.658088 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0924 16:07:03.658192 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0924 16:07:03.660218 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:07:03.661167 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:07:03.661180 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:07:03.661329 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:07:03.662304 (+ 975us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:07:03.662307 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:07:03.662361 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:07:03.662491 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:07:03.662957 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:07:03.662970 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:07:03.662979 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 16:07:03.663326 (+ 347us) client_negotiation.cc:240] Negotiation successful
0924 16:07:03.663789 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":319,"threads_started":1}
I20260924 16:07:03.667219 14142 heartbeater.cc:347] Connected to a master server at 127.3.214.254:33899
I20260924 16:07:03.667613 14142 heartbeater.cc:464] Registering TS with master...
I20260924 16:07:03.668398 14142 heartbeater.cc:511] Master 127.3.214.254:33899 requested a full tablet report, sending...
I20260924 16:07:03.671159 13955 ts_manager.cc:194] Registered new tserver with Master: 96a8c3d0804043a095452ee4468154c5 (127.3.214.193:33077)
W20260924 16:07:04.073776 14146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:07:04.074195 14146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:07:04.074331 14146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:07:04.082855 14146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:07:04.083002 14146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:07:04.083142 14146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:07:04.094182 14146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:33899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:07:04.096081 14146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:07:04.098681 14146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:07:04.115751 14151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:07:04.116096 14152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:07:04.116261 14154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:07:04.118018 14146 server_base.cc:1034] running on GCE node
I20260924 16:07:04.118949 14146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:07:04.120347 14146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:07:04.121672 14146 hybrid_clock.cc:648] HybridClock initialized: now 1790266024121604 us; error 52 us; skew 500 ppm
I20260924 16:07:04.122161 14146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:07:04.125386 14146 webserver.cc:460] Webserver started at http://127.3.214.194:42991/ using document root <none> and password file <none>
I20260924 16:07:04.126415 14146 fs_manager.cc:359] Metadata directory not provided
I20260924 16:07:04.126549 14146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:07:04.126962 14146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:07:04.130612 14146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "15f9ea636ac8458a98fed0411f34247e"
format_stamp: "Formatted at 2026-09-24 16:07:04 on dist-test-slave-1gc3"
server_key: "b6df9c36cf6db5bb37e20d4b4e1bd860"
server_key_iv: "a99131a0e71a43c64ed3f47555e144ee"
server_key_version: "encryptionkey@0"
I20260924 16:07:04.131515 14146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "15f9ea636ac8458a98fed0411f34247e"
format_stamp: "Formatted at 2026-09-24 16:07:04 on dist-test-slave-1gc3"
server_key: "b6df9c36cf6db5bb37e20d4b4e1bd860"
server_key_iv: "a99131a0e71a43c64ed3f47555e144ee"
server_key_version: "encryptionkey@0"
I20260924 16:07:04.138262 14146 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 16:07:04.143018 14160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:04.144928 14146 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:07:04.145177 14146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "15f9ea636ac8458a98fed0411f34247e"
format_stamp: "Formatted at 2026-09-24 16:07:04 on dist-test-slave-1gc3"
server_key: "b6df9c36cf6db5bb37e20d4b4e1bd860"
server_key_iv: "a99131a0e71a43c64ed3f47555e144ee"
server_key_version: "encryptionkey@0"
I20260924 16:07:04.145457 14146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:07:04.152652 14146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:07:04.153748 14146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:07:04.154050 14146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:07:04.155282 14146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:07:04.157159 14146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:07:04.157264 14146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:04.157388 14146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:07:04.157462 14146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:04.199934 14146 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:41185
I20260924 16:07:04.199990 14273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:41185 every 8 connection(s)
I20260924 16:07:04.201570 14146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:07:04.204424 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 14146
I20260924 16:07:04.204666 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:07:04.205303 3931 external_mini_cluster.cc:1428] Setting key 9cf5b61ce5479f911dc827616431f24a
I20260924 16:07:04.209146 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:33899
--builtin_ntp_servers=127.3.214.212:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 16:07:04.212556 14276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:04.204978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:45189 (local address 127.3.214.254:33899)
0924 16:07:04.205499 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:04.205505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:04.207755 (+ 2250us) server_negotiation.cc:407] Connection header received
0924 16:07:04.209944 (+ 2189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:04.209949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:04.210023 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:04.210144 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:04.211601 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:07:04.211607 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:07:04.211612 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:07:04.211632 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:07:04.211680 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:07:04.211693 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:07:04.211833 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:04.212037 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 16:07:04.212117 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":227,"threads_started":1}
I20260924 16:07:04.213680 14277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:04.205985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:33899 (local address 127.3.214.194:45189)
0924 16:07:04.207544 (+ 1559us) negotiation.cc:107] Waiting for socket to connect
0924 16:07:04.207623 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0924 16:07:04.209617 (+ 1994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:07:04.210444 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:07:04.210455 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:07:04.210564 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:07:04.211261 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:07:04.211264 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:07:04.211307 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:07:04.211417 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:07:04.211783 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:07:04.211795 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:07:04.211804 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 16:07:04.212104 (+ 300us) client_negotiation.cc:240] Negotiation successful
0924 16:07:04.212522 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":374,"threads_started":1}
I20260924 16:07:04.215462 14274 heartbeater.cc:347] Connected to a master server at 127.3.214.254:33899
I20260924 16:07:04.215804 14274 heartbeater.cc:464] Registering TS with master...
I20260924 16:07:04.216516 14274 heartbeater.cc:511] Master 127.3.214.254:33899 requested a full tablet report, sending...
I20260924 16:07:04.218143 13955 ts_manager.cc:194] Registered new tserver with Master: 15f9ea636ac8458a98fed0411f34247e (127.3.214.194:41185)
W20260924 16:07:04.634928 14278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:07:04.635313 14278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:07:04.635438 14278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:07:04.643945 14278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:07:04.644101 14278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:07:04.644268 14278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:07:04.658012 14278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:33899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:07:04.659865 14278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:07:04.662132 14278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:07:04.674474 14142 heartbeater.cc:503] Master 127.3.214.254:33899 was elected leader, sending a full tablet report...
W20260924 16:07:04.677594 14283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:07:04.677629 14286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:07:04.678267 14284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:07:04.679592 14278 server_base.cc:1034] running on GCE node
I20260924 16:07:04.680402 14278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:07:04.681723 14278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:07:04.682952 14278 hybrid_clock.cc:648] HybridClock initialized: now 1790266024682870 us; error 70 us; skew 500 ppm
I20260924 16:07:04.683342 14278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:07:04.691654 14278 webserver.cc:460] Webserver started at http://127.3.214.195:44215/ using document root <none> and password file <none>
I20260924 16:07:04.692667 14278 fs_manager.cc:359] Metadata directory not provided
I20260924 16:07:04.692780 14278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:07:04.693141 14278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:07:04.696851 14278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "5aea8b6a233a4bb58627f6a610700749"
format_stamp: "Formatted at 2026-09-24 16:07:04 on dist-test-slave-1gc3"
server_key: "6333bcfebf9ba39f56a4686dbdbb8c05"
server_key_iv: "290cb76026da9ea2cc02d6022816744a"
server_key_version: "encryptionkey@0"
I20260924 16:07:04.697765 14278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "5aea8b6a233a4bb58627f6a610700749"
format_stamp: "Formatted at 2026-09-24 16:07:04 on dist-test-slave-1gc3"
server_key: "6333bcfebf9ba39f56a4686dbdbb8c05"
server_key_iv: "290cb76026da9ea2cc02d6022816744a"
server_key_version: "encryptionkey@0"
I20260924 16:07:04.704284 14278 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260924 16:07:04.709072 14292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:04.711263 14278 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:07:04.711504 14278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "5aea8b6a233a4bb58627f6a610700749"
format_stamp: "Formatted at 2026-09-24 16:07:04 on dist-test-slave-1gc3"
server_key: "6333bcfebf9ba39f56a4686dbdbb8c05"
server_key_iv: "290cb76026da9ea2cc02d6022816744a"
server_key_version: "encryptionkey@0"
I20260924 16:07:04.711797 14278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:07:04.719285 14278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:07:04.720400 14278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:07:04.720712 14278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:07:04.721894 14278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:07:04.723794 14278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:07:04.723891 14278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:04.724026 14278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:07:04.724107 14278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:04.761744 14278 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:39403
I20260924 16:07:04.761797 14405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:39403 every 8 connection(s)
I20260924 16:07:04.763450 14278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:07:04.772701 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 14278
I20260924 16:07:04.772936 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:07:04.773624 3931 external_mini_cluster.cc:1428] Setting key 491996d495b189b57c8e42479791a62f
I20260924 16:07:04.774091 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:04.766756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:56929 (local address 127.3.214.254:33899)
0924 16:07:04.767407 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:04.767416 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:04.769111 (+ 1695us) server_negotiation.cc:407] Connection header received
0924 16:07:04.771184 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:04.771192 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:04.771281 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:04.771434 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:04.773117 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:07:04.773125 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:07:04.773131 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:07:04.773159 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:07:04.773231 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:07:04.773251 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:07:04.773424 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:04.773587 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 16:07:04.773678 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":290,"threads_started":1}
I20260924 16:07:04.775023 14409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:04.767646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:33899 (local address 127.3.214.195:56929)
0924 16:07:04.768885 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0924 16:07:04.768937 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 16:07:04.770819 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:07:04.771732 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:07:04.771742 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:07:04.771860 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:07:04.772713 (+ 853us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:07:04.772715 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:07:04.772758 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:07:04.772885 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:07:04.773370 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:07:04.773378 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:07:04.773384 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 16:07:04.773636 (+ 252us) client_negotiation.cc:240] Negotiation successful
0924 16:07:04.773906 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":327,"threads_started":1}
I20260924 16:07:04.777050 14406 heartbeater.cc:347] Connected to a master server at 127.3.214.254:33899
I20260924 16:07:04.777415 14406 heartbeater.cc:464] Registering TS with master...
I20260924 16:07:04.778142 14406 heartbeater.cc:511] Master 127.3.214.254:33899 requested a full tablet report, sending...
I20260924 16:07:04.779712 13954 ts_manager.cc:194] Registered new tserver with Master: 5aea8b6a233a4bb58627f6a610700749 (127.3.214.195:39403)
I20260924 16:07:04.791064 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 16:07:04.798310 14408 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:07:04.796571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40686 (local address 127.3.214.254:33899)
0924 16:07:04.796804 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:04.796809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:04.797263 (+ 454us) server_negotiation.cc:407] Connection header received
0924 16:07:04.797371 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:04.797375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:04.797470 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:04.797611 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:04.797975 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40686: BlockingRecv error: recv got EOF from 127.0.0.1:40686 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:07:04.798668 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 14014
I20260924 16:07:04.803663 14416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:04.800675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.3.214.193:33077)
0924 16:07:04.801269 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:04.801277 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:04.801309 (+ 32us) server_negotiation.cc:407] Connection header received
0924 16:07:04.801385 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:04.801392 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:04.801632 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:04.801784 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:04.802859 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:07:04.802866 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:07:04.802879 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:07:04.802892 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:07:04.802942 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:07:04.802958 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:07:04.803119 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:04.803245 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 16:07:04.803343 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":271,"threads_started":1}
I20260924 16:07:04.804996 14137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:07:04.894865 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 14014
I20260924 16:07:04.910983 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 14146
I20260924 16:07:04.915210 14418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:04.911965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40692 (local address 127.3.214.194:41185)
0924 16:07:04.912643 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:04.912653 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:04.912695 (+ 42us) server_negotiation.cc:407] Connection header received
0924 16:07:04.912779 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:04.912787 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:04.913084 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:04.913261 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:04.914416 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:07:04.914423 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:07:04.914438 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:07:04.914450 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:07:04.914502 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:07:04.914520 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:07:04.914678 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:04.914769 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 16:07:04.914902 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":286,"threads_started":1}
I20260924 16:07:04.916680 14269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:07:05.003058 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 14146
I20260924 16:07:05.020061 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 14278
I20260924 16:07:05.025580 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:05.021359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53690 (local address 127.3.214.195:39403)
0924 16:07:05.022336 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:05.022353 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:05.022405 (+ 52us) server_negotiation.cc:407] Connection header received
0924 16:07:05.022507 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:05.022518 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:05.022905 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:05.023125 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:05.024515 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:07:05.024526 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:07:05.024545 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:07:05.024569 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:07:05.024632 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:07:05.024653 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:07:05.024843 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:05.024995 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 16:07:05.025143 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":414,"threads_started":1}
I20260924 16:07:05.027441 14401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:07:05.114042 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 14278
I20260924 16:07:05.129659 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 13923
I20260924 16:07:05.131170 13985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:07:05.247216 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 13923
2026-09-24T16:07:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2828 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:07:05 dist-test-slave-1gc3 krb5kdc[14425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:07:05 dist-test-slave-1gc3 krb5kdc[14425](info): set up 2 sockets
Sep 24 16:07:05 dist-test-slave-1gc3 krb5kdc[14425](info): commencing operation
krb5kdc: starting...
W20260924 16:07:06.304621 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:07:06 dist-test-slave-1gc3 krb5kdc[14425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:07:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:07:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:07:06.590164 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:34851
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:39435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:34851
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:07:06.976285 14438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:07:06.976625 14438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:07:06.976753 14438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:07:06.984943 14438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:07:06.985097 14438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:07:06.985155 14438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:07:06.985262 14438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:07:06.985337 14438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:07:06.996022 14438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:39435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:34851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:34851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:07:06.997505 14438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:07:06.999014 14438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:07:07.012292 14443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:07:07.012516 14444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:07:07.012557 14446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:07:07.014132 14438 server_base.cc:1034] running on GCE node
I20260924 16:07:07.015309 14438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:07:07.017379 14438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:07:07.018821 14438 hybrid_clock.cc:648] HybridClock initialized: now 1790266027018704 us; error 82 us; skew 500 ppm
I20260924 16:07:07.019393 14438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:07:07 dist-test-slave-1gc3 krb5kdc[14425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266027, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:07:07.023813 14438 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:07:07.026141 14438 webserver.cc:460] Webserver started at http://127.3.214.254:40251/ using document root <none> and password file <none>
I20260924 16:07:07.027169 14438 fs_manager.cc:359] Metadata directory not provided
I20260924 16:07:07.027335 14438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:07:07.027719 14438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:07:07.031776 14438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "1fb9d0425574463a86b9e5bb0827bc5c"
format_stamp: "Formatted at 2026-09-24 16:07:07 on dist-test-slave-1gc3"
server_key: "cc10da70a4614668938263657f02e45d"
server_key_iv: "e0d93d5c07cddf4590f252569a1b8516"
server_key_version: "encryptionkey@0"
I20260924 16:07:07.032778 14438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "1fb9d0425574463a86b9e5bb0827bc5c"
format_stamp: "Formatted at 2026-09-24 16:07:07 on dist-test-slave-1gc3"
server_key: "cc10da70a4614668938263657f02e45d"
server_key_iv: "e0d93d5c07cddf4590f252569a1b8516"
server_key_version: "encryptionkey@0"
I20260924 16:07:07.039793 14438 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260924 16:07:07.044544 14453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:07.046485 14438 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:07:07.046723 14438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "1fb9d0425574463a86b9e5bb0827bc5c"
format_stamp: "Formatted at 2026-09-24 16:07:07 on dist-test-slave-1gc3"
server_key: "cc10da70a4614668938263657f02e45d"
server_key_iv: "e0d93d5c07cddf4590f252569a1b8516"
server_key_version: "encryptionkey@0"
I20260924 16:07:07.046990 14438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:07:07.082528 14438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:07:07.088718 14438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:07:07.089074 14438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:07:07.112125 14438 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:34851
I20260924 16:07:07.112191 14504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:34851 every 8 connection(s)
I20260924 16:07:07.114019 14438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:07:07.116461 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 14438
I20260924 16:07:07.116703 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:07:07.117448 3931 external_mini_cluster.cc:1428] Setting key e63af05a8e4b6c42b9a8494f5528ce77
I20260924 16:07:07.118824 14505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 16:07:07 dist-test-slave-1gc3 krb5kdc[14425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:07:07.133252 14505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c: Bootstrap starting.
I20260924 16:07:07.137338 14505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c: Neither blocks nor log segments found. Creating new log.
I20260924 16:07:07.138612 14505 log.cc:824] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c: Log is configured to *not* fsync() on all Append() calls
I20260924 16:07:07.142616 14505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c: No bootstrap required, opened a new log
I20260924 16:07:07.144088 14508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:07.119886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55048 (local address 127.3.214.254:34851)
0924 16:07:07.120901 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:07.120919 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:07.120975 (+ 56us) server_negotiation.cc:407] Connection header received
0924 16:07:07.122924 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:07.122955 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:07.123533 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:07.124026 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:07.125706 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:07.127303 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:07.128621 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:07.129485 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:07.129688 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:07.134755 (+ 5067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:07:07.134766 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:07:07.134788 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:07:07.134849 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:07:07.138602 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:07:07.139714 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:07:07.139725 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:07:07.139745 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:07:07.140046 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:07:07.140473 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:07:07.140477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:07:07.140479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:07:07.141215 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:07:07.141520 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:07.142060 (+ 540us) server_negotiation.cc:299] Negotiation successful
0924 16:07:07.142699 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":421,"threads_started":1}
I20260924 16:07:07.150933 14505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb9d0425574463a86b9e5bb0827bc5c" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34851 } }
I20260924 16:07:07.151404 14505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:07:07.151614 14505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fb9d0425574463a86b9e5bb0827bc5c, State: Initialized, Role: FOLLOWER
I20260924 16:07:07.152323 14505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb9d0425574463a86b9e5bb0827bc5c" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34851 } }
I20260924 16:07:07.152587 14505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:07:07.152765 14505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:07:07.152935 14505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:07:07.155311 14505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb9d0425574463a86b9e5bb0827bc5c" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34851 } }
I20260924 16:07:07.155884 14505 leader_election.cc:302] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fb9d0425574463a86b9e5bb0827bc5c; no voters:
I20260924 16:07:07.156561 14505 leader_election.cc:288] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:07:07.156754 14510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:07:07.157744 14510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c [term 1 LEADER]: Becoming Leader. State: Replica: 1fb9d0425574463a86b9e5bb0827bc5c, State: Running, Role: LEADER
I20260924 16:07:07.158502 14510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb9d0425574463a86b9e5bb0827bc5c" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34851 } }
I20260924 16:07:07.159144 14505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:07:07.161934 14511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fb9d0425574463a86b9e5bb0827bc5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb9d0425574463a86b9e5bb0827bc5c" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34851 } } }
I20260924 16:07:07.162554 14511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c [sys.catalog]: This master's current role is: LEADER
I20260924 16:07:07.161962 14512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fb9d0425574463a86b9e5bb0827bc5c. Latest consensus state: current_term: 1 leader_uuid: "1fb9d0425574463a86b9e5bb0827bc5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb9d0425574463a86b9e5bb0827bc5c" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34851 } } }
I20260924 16:07:07.164052 14512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c [sys.catalog]: This master's current role is: LEADER
I20260924 16:07:07.165759 14517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:07:07.171643 14517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:07:07.186161 14517 catalog_manager.cc:1347] Generated new cluster ID: 0bbb7325763a41bd8a15c0b4b1fb6b5c
I20260924 16:07:07.186345 14517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:07:07.207347 14517 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:07:07.209605 14517 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:07:07.219110 14517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c: Generated new TSK 0
I20260924 16:07:07.220505 14517 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.214.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
I20260924 16:07:07.298851 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:34851
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:39435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:07:07.655337 14531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:07:07.655659 14531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:07:07.655784 14531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:07:07.663581 14531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:07:07.663714 14531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:07:07.663866 14531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:07:07.674039 14531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:39435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:34851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:07:07.675439 14531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:07:07.676846 14531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:07:07.691203 14537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:07:07.691301 14536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:07:07.693260 14539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:07:07.693684 14531 server_base.cc:1034] running on GCE node
I20260924 16:07:07.694664 14531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:07:07.696048 14531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:07:07.697321 14531 hybrid_clock.cc:648] HybridClock initialized: now 1790266027697255 us; error 48 us; skew 500 ppm
I20260924 16:07:07.697750 14531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:07:07 dist-test-slave-1gc3 krb5kdc[14425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266027, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:07:07.701699 14531 init.cc:377] Logged in from keytab as kudu/127.3.214.193@KRBTEST.COM (short username kudu)
I20260924 16:07:07.703608 14531 webserver.cc:460] Webserver started at http://127.3.214.193:35051/ using document root <none> and password file <none>
I20260924 16:07:07.704566 14531 fs_manager.cc:359] Metadata directory not provided
I20260924 16:07:07.704698 14531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:07:07.705055 14531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:07:07.708644 14531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "b356bd8bd39b41ac89b9a9f9f24a9ee9"
format_stamp: "Formatted at 2026-09-24 16:07:07 on dist-test-slave-1gc3"
server_key: "187fb02b311d571276a56f72bbbf5706"
server_key_iv: "7b329ce82e896a89693330ddcb3c46ff"
server_key_version: "encryptionkey@0"
I20260924 16:07:07.709549 14531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "b356bd8bd39b41ac89b9a9f9f24a9ee9"
format_stamp: "Formatted at 2026-09-24 16:07:07 on dist-test-slave-1gc3"
server_key: "187fb02b311d571276a56f72bbbf5706"
server_key_iv: "7b329ce82e896a89693330ddcb3c46ff"
server_key_version: "encryptionkey@0"
I20260924 16:07:07.716200 14531 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260924 16:07:07.721186 14546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:07.723363 14531 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:07:07.723664 14531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "b356bd8bd39b41ac89b9a9f9f24a9ee9"
format_stamp: "Formatted at 2026-09-24 16:07:07 on dist-test-slave-1gc3"
server_key: "187fb02b311d571276a56f72bbbf5706"
server_key_iv: "7b329ce82e896a89693330ddcb3c46ff"
server_key_version: "encryptionkey@0"
I20260924 16:07:07.723950 14531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:07:07.746680 14531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:07:07.751538 14531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:07:07.751834 14531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:07:07.753119 14531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:07:07.754771 14531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:07:07.754853 14531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:07.754967 14531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:07:07.755018 14531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:07.791339 14531 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:45855
I20260924 16:07:07.791428 14660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:45855 every 8 connection(s)
I20260924 16:07:07.792943 14531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:07:07.793522 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 14531
I20260924 16:07:07.793754 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:07:07.794626 3931 external_mini_cluster.cc:1428] Setting key 32559a011b377d385c8f455891957d2c
Sep 24 16:07:07 dist-test-slave-1gc3 krb5kdc[14425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266027, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:07:07.819159 14663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:07.796169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:42939 (local address 127.3.214.254:34851)
0924 16:07:07.796650 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:07.796655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:07.798261 (+ 1606us) server_negotiation.cc:407] Connection header received
0924 16:07:07.800994 (+ 2733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:07.801000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:07.801088 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:07.801227 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:07.803445 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:07.804275 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:07.806581 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:07.807059 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:07.807186 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:07.811366 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:07:07.811374 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:07:07.811380 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:07:07.811441 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:07:07.815028 (+ 3587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:07:07.816106 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:07:07.816113 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:07:07.816116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:07:07.816315 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:07:07.816895 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:07:07.816899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:07:07.816902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:07:07.817337 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:07:07.817520 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:07.818390 (+ 870us) server_negotiation.cc:299] Negotiation successful
0924 16:07:07.818722 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":257,"threads_started":1}
I20260924 16:07:07.819885 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:07.796985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:34851 (local address 127.3.214.193:42939)
0924 16:07:07.798037 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0924 16:07:07.798126 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0924 16:07:07.800633 (+ 2507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:07:07.801536 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:07:07.801549 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:07:07.802333 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:07:07.803194 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:07:07.803221 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:07:07.804469 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:07:07.804475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:07:07.806394 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:07:07.806411 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:07:07.807339 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:07:07.807347 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:07:07.807652 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:07.808591 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:07:07.808630 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:07:07.810976 (+ 2346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:07:07.815278 (+ 4302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:07:07.815289 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:07:07.815320 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:07:07.815925 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:07:07.816489 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:07:07.816494 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:07:07.816496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:07:07.816743 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:07:07.817492 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:07:07.817499 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:07:07.817897 (+ 398us) client_negotiation.cc:769] Sending connection context
0924 16:07:07.818220 (+ 323us) client_negotiation.cc:240] Negotiation successful
0924 16:07:07.818718 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":352,"threads_started":1}
I20260924 16:07:07.821714 14661 heartbeater.cc:347] Connected to a master server at 127.3.214.254:34851
I20260924 16:07:07.822014 14661 heartbeater.cc:464] Registering TS with master...
I20260924 16:07:07.822846 14661 heartbeater.cc:511] Master 127.3.214.254:34851 requested a full tablet report, sending...
I20260924 16:07:07.825882 14470 ts_manager.cc:194] Registered new tserver with Master: b356bd8bd39b41ac89b9a9f9f24a9ee9 (127.3.214.193:45855)
WARNING: no policy specified for kudu/127.3.214.194@KRBTEST.COM; defaulting to no policy
I20260924 16:07:07.828522 14470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:42939
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
I20260924 16:07:07.863968 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:34851
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:39435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:07:08.274400 14667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:07:08.274785 14667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:07:08.274895 14667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:07:08.282886 14667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:07:08.283035 14667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:07:08.283166 14667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:07:08.294409 14667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:39435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:34851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:07:08.296149 14667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:07:08.297818 14667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:07:08.313791 14672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:07:08.314059 14673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:07:08.314035 14675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:07:08.314754 14667 server_base.cc:1034] running on GCE node
I20260924 16:07:08.315515 14667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:07:08.316730 14667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:07:08.318024 14667 hybrid_clock.cc:648] HybridClock initialized: now 1790266028317933 us; error 56 us; skew 500 ppm
I20260924 16:07:08.318495 14667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:07:08 dist-test-slave-1gc3 krb5kdc[14425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266028, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:07:08.322583 14667 init.cc:377] Logged in from keytab as kudu/127.3.214.194@KRBTEST.COM (short username kudu)
I20260924 16:07:08.324647 14667 webserver.cc:460] Webserver started at http://127.3.214.194:40407/ using document root <none> and password file <none>
I20260924 16:07:08.325584 14667 fs_manager.cc:359] Metadata directory not provided
I20260924 16:07:08.325737 14667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:07:08.326174 14667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:07:08.330080 14667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "1e38f881b891438c86668cc8d0aea68a"
format_stamp: "Formatted at 2026-09-24 16:07:08 on dist-test-slave-1gc3"
server_key: "6c86682b68d604dc70d3d1548e4a0051"
server_key_iv: "743f0276ad80576757cdcd4f274844c7"
server_key_version: "encryptionkey@0"
I20260924 16:07:08.331122 14667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "1e38f881b891438c86668cc8d0aea68a"
format_stamp: "Formatted at 2026-09-24 16:07:08 on dist-test-slave-1gc3"
server_key: "6c86682b68d604dc70d3d1548e4a0051"
server_key_iv: "743f0276ad80576757cdcd4f274844c7"
server_key_version: "encryptionkey@0"
I20260924 16:07:08.338188 14667 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260924 16:07:08.343130 14683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:08.345177 14667 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:07:08.345412 14667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "1e38f881b891438c86668cc8d0aea68a"
format_stamp: "Formatted at 2026-09-24 16:07:08 on dist-test-slave-1gc3"
server_key: "6c86682b68d604dc70d3d1548e4a0051"
server_key_iv: "743f0276ad80576757cdcd4f274844c7"
server_key_version: "encryptionkey@0"
I20260924 16:07:08.345690 14667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:07:08.363809 14667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:07:08.369719 14667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:07:08.370239 14667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:07:08.371613 14667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:07:08.373462 14667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:07:08.373566 14667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:08.373710 14667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:07:08.373777 14667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:08.411407 14667 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:46483
I20260924 16:07:08.411464 14796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:46483 every 8 connection(s)
I20260924 16:07:08.413014 14667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:07:08.416277 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 14667
I20260924 16:07:08.416498 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:07:08.417061 3931 external_mini_cluster.cc:1428] Setting key 46ac420142fc2ef65af9fb7ea4602a7b
Sep 24 16:07:08 dist-test-slave-1gc3 krb5kdc[14425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266028, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:07:08.436200 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:08.416323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:52701 (local address 127.3.214.254:34851)
0924 16:07:08.416976 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:08.416983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:08.418229 (+ 1246us) server_negotiation.cc:407] Connection header received
0924 16:07:08.420463 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:08.420469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:08.420567 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:08.420678 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:08.422612 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:08.423432 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:08.425131 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:08.425596 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:08.425715 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:08.429781 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:07:08.429786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:07:08.429790 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:07:08.429831 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:07:08.432661 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:07:08.433509 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:07:08.433513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:07:08.433516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:07:08.433681 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:07:08.434177 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:07:08.434181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:07:08.434183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:07:08.434506 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:07:08.434653 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:08.435494 (+ 841us) server_negotiation.cc:299] Negotiation successful
0924 16:07:08.435760 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":289,"threads_started":1}
I20260924 16:07:08.437502 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:08.416977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:34851 (local address 127.3.214.194:52701)
0924 16:07:08.417986 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0924 16:07:08.418061 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0924 16:07:08.420124 (+ 2063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:07:08.420985 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:07:08.420995 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:07:08.421641 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:07:08.422414 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:07:08.422438 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:07:08.423587 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:07:08.423601 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:07:08.424973 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:07:08.424988 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:07:08.425843 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:07:08.425851 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:07:08.426144 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:08.426946 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:07:08.426972 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:07:08.429503 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:07:08.432831 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:07:08.432844 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:07:08.432864 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:07:08.433374 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:07:08.433792 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:07:08.433797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:07:08.433801 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:07:08.434020 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:07:08.434644 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:07:08.434653 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:07:08.435089 (+ 436us) client_negotiation.cc:769] Sending connection context
0924 16:07:08.435372 (+ 283us) client_negotiation.cc:240] Negotiation successful
0924 16:07:08.436073 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":382,"threads_started":1}
I20260924 16:07:08.439456 14797 heartbeater.cc:347] Connected to a master server at 127.3.214.254:34851
I20260924 16:07:08.440011 14797 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.214.195@KRBTEST.COM; defaulting to no policy
I20260924 16:07:08.441018 14797 heartbeater.cc:511] Master 127.3.214.254:34851 requested a full tablet report, sending...
I20260924 16:07:08.442847 14470 ts_manager.cc:194] Registered new tserver with Master: 1e38f881b891438c86668cc8d0aea68a (127.3.214.194:46483)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.195@KRBTEST.COM" created.
I20260924 16:07:08.443760 14470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:52701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
I20260924 16:07:08.474345 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:34851
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:39435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:07:08.832487 14661 heartbeater.cc:503] Master 127.3.214.254:34851 was elected leader, sending a full tablet report...
W20260924 16:07:08.853888 14803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:07:08.854269 14803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:07:08.854420 14803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:07:08.862300 14803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:07:08.862458 14803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:07:08.862646 14803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:07:08.873553 14803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:39435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:34851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:07:08.875090 14803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:07:08.876539 14803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:07:08.892125 14808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:07:08.892246 14809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:07:08.892170 14811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:07:08.893553 14803 server_base.cc:1034] running on GCE node
I20260924 16:07:08.894451 14803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:07:08.895622 14803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:07:08.896889 14803 hybrid_clock.cc:648] HybridClock initialized: now 1790266028896797 us; error 79 us; skew 500 ppm
I20260924 16:07:08.897281 14803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:07:08 dist-test-slave-1gc3 krb5kdc[14425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266028, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:07:08.901055 14803 init.cc:377] Logged in from keytab as kudu/127.3.214.195@KRBTEST.COM (short username kudu)
I20260924 16:07:08.903163 14803 webserver.cc:460] Webserver started at http://127.3.214.195:35331/ using document root <none> and password file <none>
I20260924 16:07:08.903993 14803 fs_manager.cc:359] Metadata directory not provided
I20260924 16:07:08.904130 14803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:07:08.904490 14803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:07:08.907862 14803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "914c8e0fa017432f83b7e64e82db50b6"
format_stamp: "Formatted at 2026-09-24 16:07:08 on dist-test-slave-1gc3"
server_key: "acc4c3fb797ac4a9d6b35304c19d1fad"
server_key_iv: "401cd3030e5babbcc8c39c25feb60e9c"
server_key_version: "encryptionkey@0"
I20260924 16:07:08.908730 14803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "914c8e0fa017432f83b7e64e82db50b6"
format_stamp: "Formatted at 2026-09-24 16:07:08 on dist-test-slave-1gc3"
server_key: "acc4c3fb797ac4a9d6b35304c19d1fad"
server_key_iv: "401cd3030e5babbcc8c39c25feb60e9c"
server_key_version: "encryptionkey@0"
I20260924 16:07:08.915339 14803 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260924 16:07:08.919801 14818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:08.921581 14803 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:07:08.921816 14803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "914c8e0fa017432f83b7e64e82db50b6"
format_stamp: "Formatted at 2026-09-24 16:07:08 on dist-test-slave-1gc3"
server_key: "acc4c3fb797ac4a9d6b35304c19d1fad"
server_key_iv: "401cd3030e5babbcc8c39c25feb60e9c"
server_key_version: "encryptionkey@0"
I20260924 16:07:08.922039 14803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:07:08.936002 14803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:07:08.940783 14803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:07:08.941110 14803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:07:08.942279 14803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:07:08.943821 14803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:07:08.943910 14803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:08.944043 14803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:07:08.944104 14803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:07:08.980798 14803 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:41497
I20260924 16:07:08.980896 14931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:41497 every 8 connection(s)
I20260924 16:07:08.982262 14803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:07:08.984951 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 14803
I20260924 16:07:08.985219 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:07:08.985829 3931 external_mini_cluster.cc:1428] Setting key 86eee9d15350ee83fc99792eebb73587
Sep 24 16:07:08 dist-test-slave-1gc3 krb5kdc[14425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266028, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:07:09.007639 14934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:08.985794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:40243 (local address 127.3.214.254:34851)
0924 16:07:08.986527 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:08.986536 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:08.987847 (+ 1311us) server_negotiation.cc:407] Connection header received
0924 16:07:08.990461 (+ 2614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:08.990465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:08.990550 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:08.990661 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:08.992562 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:08.993403 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:08.995221 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:08.995662 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:08.995788 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:08.999940 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:07:08.999947 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:07:08.999951 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:07:09.000000 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:07:09.003540 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:07:09.004695 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:07:09.004701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:07:09.004705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:07:09.004871 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:07:09.005498 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:07:09.005503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:07:09.005505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:07:09.005823 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:07:09.005974 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:09.007070 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0924 16:07:09.007294 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":333,"threads_started":1}
I20260924 16:07:09.009037 14935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:08.986428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:34851 (local address 127.3.214.195:40243)
0924 16:07:08.987568 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0924 16:07:08.987724 (+ 156us) client_negotiation.cc:174] Beginning negotiation
0924 16:07:08.990082 (+ 2358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:07:08.990918 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:07:08.990929 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:07:08.991587 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:07:08.992342 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:07:08.992364 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:07:08.993561 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:07:08.993565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:07:08.995041 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:07:08.995056 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:07:08.995893 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:07:08.995898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:07:08.996136 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:08.996896 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:07:08.996922 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:07:08.999553 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:07:09.003749 (+ 4196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:07:09.003762 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:07:09.003785 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:07:09.004496 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:07:09.005045 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:07:09.005050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:07:09.005052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:07:09.005329 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:07:09.005985 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:07:09.005994 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:07:09.006606 (+ 612us) client_negotiation.cc:769] Sending connection context
0924 16:07:09.006918 (+ 312us) client_negotiation.cc:240] Negotiation successful
0924 16:07:09.007536 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"thread_start_us":438,"threads_started":1}
I20260924 16:07:09.011059 14932 heartbeater.cc:347] Connected to a master server at 127.3.214.254:34851
I20260924 16:07:09.011492 14932 heartbeater.cc:464] Registering TS with master...
I20260924 16:07:09.012456 14932 heartbeater.cc:511] Master 127.3.214.254:34851 requested a full tablet report, sending...
I20260924 16:07:09.014576 14470 ts_manager.cc:194] Registered new tserver with Master: 914c8e0fa017432f83b7e64e82db50b6 (127.3.214.195:41497)
I20260924 16:07:09.015642 14470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:40243
I20260924 16:07:09.016664 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:07:09.037537 14934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:09.022964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55052 (local address 127.3.214.254:34851)
0924 16:07:09.023270 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:09.023276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:09.023644 (+ 368us) server_negotiation.cc:407] Connection header received
0924 16:07:09.023848 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:09.023853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:09.023933 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:09.024038 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:09.025745 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.026668 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.028281 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.028847 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.028991 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:09.031745 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:07:09.031752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:07:09.031756 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:07:09.031804 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:07:09.034799 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:07:09.035592 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:07:09.035596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:07:09.035598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:07:09.035734 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:07:09.036057 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:07:09.036060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:07:09.036063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:07:09.036436 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:07:09.036639 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:09.037012 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 16:07:09.037224 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260924 16:07:09.042461 14469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:07:09.049764 14469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:07:09.096006 14950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:09.066355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35548 (local address 127.3.214.194:46483)
0924 16:07:09.086035 (+ 19680us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:09.086045 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:09.087339 (+ 1294us) server_negotiation.cc:407] Connection header received
0924 16:07:09.087538 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:09.087548 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:09.087879 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:09.088125 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:07:09.088972 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.090516 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.092810 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.094349 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.094501 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:09.094761 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:09.095405 (+ 644us) server_negotiation.cc:299] Negotiation successful
0924 16:07:09.095622 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18272,"thread_start_us":18094,"threads_started":1}
I20260924 16:07:09.096515 14948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:09.065588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58264 (local address 127.3.214.195:41497)
0924 16:07:09.086692 (+ 21104us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:09.086700 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:09.086743 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:07:09.086864 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:09.086873 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:09.087247 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:09.087501 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:07:09.088245 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.089629 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.093002 (+ 3373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.094656 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.094813 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:09.095077 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:09.095955 (+ 878us) server_negotiation.cc:299] Negotiation successful
0924 16:07:09.096137 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20894,"thread_start_us":20714,"threads_started":1}
I20260924 16:07:09.096654 14947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:09.065638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:46483 (local address 127.0.0.1:35548)
0924 16:07:09.087157 (+ 21519us) negotiation.cc:107] Waiting for socket to connect
0924 16:07:09.087178 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:07:09.087355 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:07:09.088136 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:07:09.088141 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:07:09.088170 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:07:09.088750 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:07:09.088764 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.090678 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.090684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:07:09.092631 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:07:09.092645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.094911 (+ 2266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.094916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:07:09.095116 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:09.095159 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 16:07:09.095321 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 16:07:09.095586 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21370,"mutex_wait_us":79,"thread_start_us":21182,"threads_started":1}
I20260924 16:07:09.096673 14949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:09.066065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34404 (local address 127.3.214.193:45855)
0924 16:07:09.082984 (+ 16919us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:09.082994 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:09.083038 (+ 44us) server_negotiation.cc:407] Connection header received
0924 16:07:09.083153 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:09.083160 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:09.083448 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:09.083715 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:07:09.086460 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.088105 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.093025 (+ 4920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.094656 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.094824 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:09.095077 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:09.096182 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0924 16:07:09.096364 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16708,"thread_start_us":17894,"threads_started":1}
I20260924 16:07:09.097369 14945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:09.065199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:45855 (local address 127.0.0.1:34404)
0924 16:07:09.078368 (+ 13169us) negotiation.cc:107] Waiting for socket to connect
0924 16:07:09.078444 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0924 16:07:09.078702 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:07:09.083764 (+ 5062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:07:09.083772 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:07:09.083811 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:07:09.085933 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:07:09.085954 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.088324 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.088330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:07:09.092641 (+ 4311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:07:09.092659 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.095288 (+ 2629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.095309 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:07:09.095605 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:09.095691 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 16:07:09.095798 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 16:07:09.095983 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12722,"mutex_wait_us":256,"thread_start_us":12558,"threads_started":1}
I20260924 16:07:09.097395 14946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:09.065228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:41497 (local address 127.0.0.1:58264)
0924 16:07:09.080185 (+ 14957us) negotiation.cc:107] Waiting for socket to connect
0924 16:07:09.080220 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 16:07:09.080470 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:07:09.087526 (+ 7056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:07:09.087531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:07:09.087556 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:07:09.087998 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:07:09.088013 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.089781 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.089787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:07:09.092638 (+ 2851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:07:09.092659 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.095332 (+ 2673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.095339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:07:09.095606 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:09.095691 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 16:07:09.095798 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 16:07:09.096887 (+ 1089us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14758,"mutex_wait_us":1057,"thread_start_us":15386,"threads_started":1}
I20260924 16:07:09.102044 14595 tablet_service.cc:1467] Processing CreateTablet for tablet fb3e2684f53a42a3bdd3173af6636ba7 (DEFAULT_TABLE table=test-table [id=eed6d36fae514171b63ef365cc087594]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:07:09.102167 14731 tablet_service.cc:1467] Processing CreateTablet for tablet fb3e2684f53a42a3bdd3173af6636ba7 (DEFAULT_TABLE table=test-table [id=eed6d36fae514171b63ef365cc087594]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:07:09.102203 14866 tablet_service.cc:1467] Processing CreateTablet for tablet fb3e2684f53a42a3bdd3173af6636ba7 (DEFAULT_TABLE table=test-table [id=eed6d36fae514171b63ef365cc087594]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:07:09.103755 14866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb3e2684f53a42a3bdd3173af6636ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:07:09.103801 14731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb3e2684f53a42a3bdd3173af6636ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:07:09.103796 14595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb3e2684f53a42a3bdd3173af6636ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:07:09.123025 14953 tablet_bootstrap.cc:492] T fb3e2684f53a42a3bdd3173af6636ba7 P 1e38f881b891438c86668cc8d0aea68a: Bootstrap starting.
I20260924 16:07:09.127063 14951 tablet_bootstrap.cc:492] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6: Bootstrap starting.
I20260924 16:07:09.129151 14952 tablet_bootstrap.cc:492] T fb3e2684f53a42a3bdd3173af6636ba7 P b356bd8bd39b41ac89b9a9f9f24a9ee9: Bootstrap starting.
I20260924 16:07:09.129974 14953 tablet_bootstrap.cc:654] T fb3e2684f53a42a3bdd3173af6636ba7 P 1e38f881b891438c86668cc8d0aea68a: Neither blocks nor log segments found. Creating new log.
I20260924 16:07:09.131445 14953 log.cc:824] T fb3e2684f53a42a3bdd3173af6636ba7 P 1e38f881b891438c86668cc8d0aea68a: Log is configured to *not* fsync() on all Append() calls
I20260924 16:07:09.133481 14951 tablet_bootstrap.cc:654] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6: Neither blocks nor log segments found. Creating new log.
I20260924 16:07:09.133610 14952 tablet_bootstrap.cc:654] T fb3e2684f53a42a3bdd3173af6636ba7 P b356bd8bd39b41ac89b9a9f9f24a9ee9: Neither blocks nor log segments found. Creating new log.
I20260924 16:07:09.135030 14951 log.cc:824] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6: Log is configured to *not* fsync() on all Append() calls
I20260924 16:07:09.135185 14952 log.cc:824] T fb3e2684f53a42a3bdd3173af6636ba7 P b356bd8bd39b41ac89b9a9f9f24a9ee9: Log is configured to *not* fsync() on all Append() calls
I20260924 16:07:09.141415 14953 tablet_bootstrap.cc:492] T fb3e2684f53a42a3bdd3173af6636ba7 P 1e38f881b891438c86668cc8d0aea68a: No bootstrap required, opened a new log
I20260924 16:07:09.141659 14951 tablet_bootstrap.cc:492] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6: No bootstrap required, opened a new log
I20260924 16:07:09.141752 14953 ts_tablet_manager.cc:1397] T fb3e2684f53a42a3bdd3173af6636ba7 P 1e38f881b891438c86668cc8d0aea68a: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.006s
I20260924 16:07:09.141819 14952 tablet_bootstrap.cc:492] T fb3e2684f53a42a3bdd3173af6636ba7 P b356bd8bd39b41ac89b9a9f9f24a9ee9: No bootstrap required, opened a new log
I20260924 16:07:09.142036 14951 ts_tablet_manager.cc:1397] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20260924 16:07:09.142200 14952 ts_tablet_manager.cc:1397] T fb3e2684f53a42a3bdd3173af6636ba7 P b356bd8bd39b41ac89b9a9f9f24a9ee9: Time spent bootstrapping tablet: real 0.013s user 0.013s sys 0.000s
I20260924 16:07:09.150065 14953 raft_consensus.cc:348] T fb3e2684f53a42a3bdd3173af6636ba7 P 1e38f881b891438c86668cc8d0aea68a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356bd8bd39b41ac89b9a9f9f24a9ee9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 45855 } } peers { permanent_uuid: "914c8e0fa017432f83b7e64e82db50b6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 41497 } } peers { permanent_uuid: "1e38f881b891438c86668cc8d0aea68a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 46483 } }
I20260924 16:07:09.150591 14953 raft_consensus.cc:374] T fb3e2684f53a42a3bdd3173af6636ba7 P 1e38f881b891438c86668cc8d0aea68a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:07:09.150712 14953 raft_consensus.cc:729] T fb3e2684f53a42a3bdd3173af6636ba7 P 1e38f881b891438c86668cc8d0aea68a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e38f881b891438c86668cc8d0aea68a, State: Initialized, Role: FOLLOWER
I20260924 16:07:09.151405 14953 consensus_queue.cc:260] T fb3e2684f53a42a3bdd3173af6636ba7 P 1e38f881b891438c86668cc8d0aea68a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356bd8bd39b41ac89b9a9f9f24a9ee9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 45855 } } peers { permanent_uuid: "914c8e0fa017432f83b7e64e82db50b6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 41497 } } peers { permanent_uuid: "1e38f881b891438c86668cc8d0aea68a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 46483 } }
I20260924 16:07:09.152815 14797 heartbeater.cc:503] Master 127.3.214.254:34851 was elected leader, sending a full tablet report...
I20260924 16:07:09.153463 14953 ts_tablet_manager.cc:1428] T fb3e2684f53a42a3bdd3173af6636ba7 P 1e38f881b891438c86668cc8d0aea68a: Time spent starting tablet: real 0.012s user 0.004s sys 0.009s
I20260924 16:07:09.153719 14951 raft_consensus.cc:348] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356bd8bd39b41ac89b9a9f9f24a9ee9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 45855 } } peers { permanent_uuid: "914c8e0fa017432f83b7e64e82db50b6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 41497 } } peers { permanent_uuid: "1e38f881b891438c86668cc8d0aea68a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 46483 } }
I20260924 16:07:09.153719 14952 raft_consensus.cc:348] T fb3e2684f53a42a3bdd3173af6636ba7 P b356bd8bd39b41ac89b9a9f9f24a9ee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356bd8bd39b41ac89b9a9f9f24a9ee9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 45855 } } peers { permanent_uuid: "914c8e0fa017432f83b7e64e82db50b6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 41497 } } peers { permanent_uuid: "1e38f881b891438c86668cc8d0aea68a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 46483 } }
I20260924 16:07:09.154343 14952 raft_consensus.cc:374] T fb3e2684f53a42a3bdd3173af6636ba7 P b356bd8bd39b41ac89b9a9f9f24a9ee9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:07:09.154343 14951 raft_consensus.cc:374] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:07:09.154467 14952 raft_consensus.cc:729] T fb3e2684f53a42a3bdd3173af6636ba7 P b356bd8bd39b41ac89b9a9f9f24a9ee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b356bd8bd39b41ac89b9a9f9f24a9ee9, State: Initialized, Role: FOLLOWER
I20260924 16:07:09.154467 14951 raft_consensus.cc:729] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914c8e0fa017432f83b7e64e82db50b6, State: Initialized, Role: FOLLOWER
I20260924 16:07:09.155284 14952 consensus_queue.cc:260] T fb3e2684f53a42a3bdd3173af6636ba7 P b356bd8bd39b41ac89b9a9f9f24a9ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356bd8bd39b41ac89b9a9f9f24a9ee9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 45855 } } peers { permanent_uuid: "914c8e0fa017432f83b7e64e82db50b6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 41497 } } peers { permanent_uuid: "1e38f881b891438c86668cc8d0aea68a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 46483 } }
I20260924 16:07:09.155287 14951 consensus_queue.cc:260] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356bd8bd39b41ac89b9a9f9f24a9ee9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 45855 } } peers { permanent_uuid: "914c8e0fa017432f83b7e64e82db50b6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 41497 } } peers { permanent_uuid: "1e38f881b891438c86668cc8d0aea68a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 46483 } }
I20260924 16:07:09.158481 14932 heartbeater.cc:503] Master 127.3.214.254:34851 was elected leader, sending a full tablet report...
I20260924 16:07:09.158087 14951 ts_tablet_manager.cc:1428] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
I20260924 16:07:09.160580 14952 ts_tablet_manager.cc:1428] T fb3e2684f53a42a3bdd3173af6636ba7 P b356bd8bd39b41ac89b9a9f9f24a9ee9: Time spent starting tablet: real 0.018s user 0.011s sys 0.004s
W20260924 16:07:09.165542 14798 tablet.cc:2369] T fb3e2684f53a42a3bdd3173af6636ba7 P 1e38f881b891438c86668cc8d0aea68a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:07:09.217978 14958 raft_consensus.cc:482] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:07:09.218433 14958 raft_consensus.cc:504] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356bd8bd39b41ac89b9a9f9f24a9ee9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 45855 } } peers { permanent_uuid: "914c8e0fa017432f83b7e64e82db50b6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 41497 } } peers { permanent_uuid: "1e38f881b891438c86668cc8d0aea68a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 46483 } }
I20260924 16:07:09.220904 14958 leader_election.cc:288] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b356bd8bd39b41ac89b9a9f9f24a9ee9 (127.3.214.193:45855), 1e38f881b891438c86668cc8d0aea68a (127.3.214.194:46483)
I20260924 16:07:09.228237 14950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:09.221438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:54441 (local address 127.3.214.194:46483)
0924 16:07:09.221672 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:09.221680 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:09.222092 (+ 412us) server_negotiation.cc:407] Connection header received
0924 16:07:09.222381 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:09.222388 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:09.222493 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:09.222612 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:07:09.223417 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.224201 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.226225 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.227244 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.227406 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:09.227492 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:09.227785 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 16:07:09.227928 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:07:09.228248 14935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:09.221690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:46483 (local address 127.3.214.195:54441)
0924 16:07:09.221936 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 16:07:09.221962 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:07:09.222155 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:07:09.222670 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:07:09.222675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:07:09.222705 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:07:09.222941 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:07:09.222954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.224354 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.224361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:07:09.226036 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:07:09.226052 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.227406 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.227413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:07:09.227613 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:09.227642 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 16:07:09.227751 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 16:07:09.227921 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":47}
I20260924 16:07:09.228610 14949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:09.221400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:45457 (local address 127.3.214.193:45855)
0924 16:07:09.221708 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:09.221715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:09.222947 (+ 1232us) server_negotiation.cc:407] Connection header received
0924 16:07:09.223088 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:09.223092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:09.223159 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:09.223254 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:07:09.223821 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.224575 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.226575 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.227531 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.227699 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:09.227803 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:09.228146 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 16:07:09.228296 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 16:07:09.228925 14960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:09.221958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:45855 (local address 127.3.214.195:45457)
0924 16:07:09.222828 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0924 16:07:09.222848 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:07:09.222976 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:07:09.223288 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:07:09.223293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:07:09.223318 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:07:09.223603 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:07:09.223611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.224746 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.224750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:07:09.226406 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:07:09.226421 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.227757 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.227763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:07:09.227947 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:09.227977 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 16:07:09.228063 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 16:07:09.228226 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":5,"thread_start_us":458,"threads_started":1}
I20260924 16:07:09.229682 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb3e2684f53a42a3bdd3173af6636ba7" candidate_uuid: "914c8e0fa017432f83b7e64e82db50b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e38f881b891438c86668cc8d0aea68a" is_pre_election: true
I20260924 16:07:09.230335 14751 raft_consensus.cc:2454] T fb3e2684f53a42a3bdd3173af6636ba7 P 1e38f881b891438c86668cc8d0aea68a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 914c8e0fa017432f83b7e64e82db50b6 in term 0.
I20260924 16:07:09.231135 14820 leader_election.cc:302] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e38f881b891438c86668cc8d0aea68a, 914c8e0fa017432f83b7e64e82db50b6; no voters:
I20260924 16:07:09.231083 14616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb3e2684f53a42a3bdd3173af6636ba7" candidate_uuid: "914c8e0fa017432f83b7e64e82db50b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b356bd8bd39b41ac89b9a9f9f24a9ee9" is_pre_election: true
I20260924 16:07:09.231583 14958 raft_consensus.cc:2781] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:07:09.231720 14958 raft_consensus.cc:482] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:07:09.231658 14616 raft_consensus.cc:2454] T fb3e2684f53a42a3bdd3173af6636ba7 P b356bd8bd39b41ac89b9a9f9f24a9ee9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 914c8e0fa017432f83b7e64e82db50b6 in term 0.
I20260924 16:07:09.231825 14958 raft_consensus.cc:3037] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6 [term 0 FOLLOWER]: Advancing to term 1
W20260924 16:07:09.234086 14933 tablet.cc:2369] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:07:09.234532 14958 raft_consensus.cc:504] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356bd8bd39b41ac89b9a9f9f24a9ee9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 45855 } } peers { permanent_uuid: "914c8e0fa017432f83b7e64e82db50b6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 41497 } } peers { permanent_uuid: "1e38f881b891438c86668cc8d0aea68a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 46483 } }
I20260924 16:07:09.235289 14958 leader_election.cc:288] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6 [CANDIDATE]: Term 1 election: Requested vote from peers b356bd8bd39b41ac89b9a9f9f24a9ee9 (127.3.214.193:45855), 1e38f881b891438c86668cc8d0aea68a (127.3.214.194:46483)
I20260924 16:07:09.235831 14616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb3e2684f53a42a3bdd3173af6636ba7" candidate_uuid: "914c8e0fa017432f83b7e64e82db50b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b356bd8bd39b41ac89b9a9f9f24a9ee9"
I20260924 16:07:09.235890 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb3e2684f53a42a3bdd3173af6636ba7" candidate_uuid: "914c8e0fa017432f83b7e64e82db50b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e38f881b891438c86668cc8d0aea68a"
I20260924 16:07:09.236049 14616 raft_consensus.cc:3037] T fb3e2684f53a42a3bdd3173af6636ba7 P b356bd8bd39b41ac89b9a9f9f24a9ee9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:07:09.236101 14751 raft_consensus.cc:3037] T fb3e2684f53a42a3bdd3173af6636ba7 P 1e38f881b891438c86668cc8d0aea68a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:07:09.238440 14616 raft_consensus.cc:2454] T fb3e2684f53a42a3bdd3173af6636ba7 P b356bd8bd39b41ac89b9a9f9f24a9ee9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 914c8e0fa017432f83b7e64e82db50b6 in term 1.
I20260924 16:07:09.238649 14751 raft_consensus.cc:2454] T fb3e2684f53a42a3bdd3173af6636ba7 P 1e38f881b891438c86668cc8d0aea68a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 914c8e0fa017432f83b7e64e82db50b6 in term 1.
I20260924 16:07:09.238931 14822 leader_election.cc:302] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 914c8e0fa017432f83b7e64e82db50b6, b356bd8bd39b41ac89b9a9f9f24a9ee9; no voters:
I20260924 16:07:09.239291 14958 raft_consensus.cc:2781] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:07:09.239830 14958 raft_consensus.cc:686] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6 [term 1 LEADER]: Becoming Leader. State: Replica: 914c8e0fa017432f83b7e64e82db50b6, State: Running, Role: LEADER
I20260924 16:07:09.240448 14958 consensus_queue.cc:237] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356bd8bd39b41ac89b9a9f9f24a9ee9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 45855 } } peers { permanent_uuid: "914c8e0fa017432f83b7e64e82db50b6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 41497 } } peers { permanent_uuid: "1e38f881b891438c86668cc8d0aea68a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 46483 } }
I20260924 16:07:09.245756 14470 catalog_manager.cc:5495] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 914c8e0fa017432f83b7e64e82db50b6 (127.3.214.195). New cstate: current_term: 1 leader_uuid: "914c8e0fa017432f83b7e64e82db50b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356bd8bd39b41ac89b9a9f9f24a9ee9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 45855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "914c8e0fa017432f83b7e64e82db50b6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 41497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e38f881b891438c86668cc8d0aea68a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 46483 } health_report { overall_health: UNKNOWN } } }
I20260924 16:07:09.265100 14948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:09.259189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58280 (local address 127.3.214.195:41497)
0924 16:07:09.259367 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:09.259372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:09.259483 (+ 111us) server_negotiation.cc:407] Connection header received
0924 16:07:09.259706 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:09.259710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:09.259815 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:09.259972 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:07:09.260362 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.261204 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.262651 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.263090 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.263197 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:09.263752 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:07:09.264443 (+ 691us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:07:09.264586 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:09.264719 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 16:07:09.264806 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:07:09.275508 14751 raft_consensus.cc:1261] T fb3e2684f53a42a3bdd3173af6636ba7 P 1e38f881b891438c86668cc8d0aea68a [term 1 FOLLOWER]: Refusing update from remote peer 914c8e0fa017432f83b7e64e82db50b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:07:09.275508 14616 raft_consensus.cc:1261] T fb3e2684f53a42a3bdd3173af6636ba7 P b356bd8bd39b41ac89b9a9f9f24a9ee9 [term 1 FOLLOWER]: Refusing update from remote peer 914c8e0fa017432f83b7e64e82db50b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:07:09.276938 14961 consensus_queue.cc:1035] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e38f881b891438c86668cc8d0aea68a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 46483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:07:09.277593 14958 consensus_queue.cc:1035] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b356bd8bd39b41ac89b9a9f9f24a9ee9" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 45855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:07:09.308965 14469 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:07:09.309298 14469 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:07:09.313859 14469 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1fb9d0425574463a86b9e5bb0827bc5c: Sending DeleteTablet for 3 replicas of tablet fb3e2684f53a42a3bdd3173af6636ba7
I20260924 16:07:09.315412 14595 tablet_service.cc:1514] Processing DeleteTablet for tablet fb3e2684f53a42a3bdd3173af6636ba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:07:09 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:34404
I20260924 16:07:09.315773 14731 tablet_service.cc:1514] Processing DeleteTablet for tablet fb3e2684f53a42a3bdd3173af6636ba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:07:09 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:35548
I20260924 16:07:09.315613 14866 tablet_service.cc:1514] Processing DeleteTablet for tablet fb3e2684f53a42a3bdd3173af6636ba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:07:09 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:58264
I20260924 16:07:09.318640 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 14531
I20260924 16:07:09.326844 14976 tablet_replica.cc:330] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6: stopping tablet replica
I20260924 16:07:09.327543 14976 raft_consensus.cc:2229] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:07:09.328501 14975 tablet_replica.cc:330] T fb3e2684f53a42a3bdd3173af6636ba7 P 1e38f881b891438c86668cc8d0aea68a: stopping tablet replica
I20260924 16:07:09.328523 14976 raft_consensus.cc:2258] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:07:09.329814 14975 raft_consensus.cc:2229] T fb3e2684f53a42a3bdd3173af6636ba7 P 1e38f881b891438c86668cc8d0aea68a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:07:09.330542 14975 raft_consensus.cc:2258] T fb3e2684f53a42a3bdd3173af6636ba7 P 1e38f881b891438c86668cc8d0aea68a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:07:09.331549 14974 tablet_replica.cc:330] T fb3e2684f53a42a3bdd3173af6636ba7 P b356bd8bd39b41ac89b9a9f9f24a9ee9: stopping tablet replica
I20260924 16:07:09.332113 14974 raft_consensus.cc:2229] T fb3e2684f53a42a3bdd3173af6636ba7 P b356bd8bd39b41ac89b9a9f9f24a9ee9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:07:09.332298 14976 ts_tablet_manager.cc:1905] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:07:09.332784 14974 raft_consensus.cc:2258] T fb3e2684f53a42a3bdd3173af6636ba7 P b356bd8bd39b41ac89b9a9f9f24a9ee9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:07:09.333130 14975 ts_tablet_manager.cc:1905] T fb3e2684f53a42a3bdd3173af6636ba7 P 1e38f881b891438c86668cc8d0aea68a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:07:09.335351 14974 ts_tablet_manager.cc:1905] T fb3e2684f53a42a3bdd3173af6636ba7 P b356bd8bd39b41ac89b9a9f9f24a9ee9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:07:09.340869 14975 ts_tablet_manager.cc:1918] T fb3e2684f53a42a3bdd3173af6636ba7 P 1e38f881b891438c86668cc8d0aea68a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:07:09.340977 14976 ts_tablet_manager.cc:1918] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:07:09.341198 14976 log.cc:1196] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/fb3e2684f53a42a3bdd3173af6636ba7
I20260924 16:07:09.341948 14976 ts_tablet_manager.cc:1939] T fb3e2684f53a42a3bdd3173af6636ba7 P 914c8e0fa017432f83b7e64e82db50b6: Deleting consensus metadata
I20260924 16:07:09.343019 14974 ts_tablet_manager.cc:1918] T fb3e2684f53a42a3bdd3173af6636ba7 P b356bd8bd39b41ac89b9a9f9f24a9ee9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:07:09.343382 14974 log.cc:1196] T fb3e2684f53a42a3bdd3173af6636ba7 P b356bd8bd39b41ac89b9a9f9f24a9ee9: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-0/wal/wals/fb3e2684f53a42a3bdd3173af6636ba7
I20260924 16:07:09.344041 14974 ts_tablet_manager.cc:1939] T fb3e2684f53a42a3bdd3173af6636ba7 P b356bd8bd39b41ac89b9a9f9f24a9ee9: Deleting consensus metadata
I20260924 16:07:09.344390 14975 log.cc:1196] T fb3e2684f53a42a3bdd3173af6636ba7 P 1e38f881b891438c86668cc8d0aea68a: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/fb3e2684f53a42a3bdd3173af6636ba7
I20260924 16:07:09.344614 14454 catalog_manager.cc:4841] TS 914c8e0fa017432f83b7e64e82db50b6 (127.3.214.195:41497): tablet fb3e2684f53a42a3bdd3173af6636ba7 (table test-table [id=eed6d36fae514171b63ef365cc087594]) successfully deleted
I20260924 16:07:09.345100 14975 ts_tablet_manager.cc:1939] T fb3e2684f53a42a3bdd3173af6636ba7 P 1e38f881b891438c86668cc8d0aea68a: Deleting consensus metadata
I20260924 16:07:09.348395 14457 catalog_manager.cc:4841] TS b356bd8bd39b41ac89b9a9f9f24a9ee9 (127.3.214.193:45855): tablet fb3e2684f53a42a3bdd3173af6636ba7 (table test-table [id=eed6d36fae514171b63ef365cc087594]) successfully deleted
I20260924 16:07:09.348593 14455 catalog_manager.cc:4841] TS 1e38f881b891438c86668cc8d0aea68a (127.3.214.194:46483): tablet fb3e2684f53a42a3bdd3173af6636ba7 (table test-table [id=eed6d36fae514171b63ef365cc087594]) successfully deleted
Sep 24 16:07:09 dist-test-slave-1gc3 krb5kdc[14425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.193@KRBTEST.COM
I20260924 16:07:09.358189 14949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:09.320204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34410 (local address 127.3.214.193:45855)
0924 16:07:09.320419 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:09.320426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:09.327729 (+ 7303us) server_negotiation.cc:407] Connection header received
0924 16:07:09.328845 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:09.328852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:09.328964 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:09.329102 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:09.341383 (+ 12281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.342645 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.344685 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.345378 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.345494 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:09.351472 (+ 5978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:07:09.351496 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:07:09.351518 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:07:09.351647 (+ 129us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:07:09.354902 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:07:09.355729 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:07:09.355739 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:07:09.355742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:07:09.355907 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:07:09.356307 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:07:09.356311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:07:09.356313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:07:09.356773 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:07:09.357098 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:09.357437 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 16:07:09.357721 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 16:07:09.359108 14656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:07:09.460670 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 14531
I20260924 16:07:09.481863 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 14667
Sep 24 16:07:09 dist-test-slave-1gc3 krb5kdc[14425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.194@KRBTEST.COM
I20260924 16:07:09.502238 14950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:09.483019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35554 (local address 127.3.214.194:46483)
0924 16:07:09.483284 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:09.483292 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:09.483320 (+ 28us) server_negotiation.cc:407] Connection header received
0924 16:07:09.483466 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:09.483472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:09.483573 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:09.483714 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:09.485102 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.486373 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.488037 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.488682 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.488837 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:09.494022 (+ 5185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:07:09.494049 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:07:09.494074 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:07:09.494171 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:07:09.498524 (+ 4353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:07:09.499656 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:07:09.499669 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:07:09.499672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:07:09.499884 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:07:09.500303 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:07:09.500309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:07:09.500312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:07:09.500901 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:07:09.501231 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:09.501518 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 16:07:09.501804 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 16:07:09.503224 14792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:07:09.605211 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 14667
I20260924 16:07:09.626315 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 14803
Sep 24 16:07:09 dist-test-slave-1gc3 krb5kdc[14425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.195@KRBTEST.COM
I20260924 16:07:09.646973 14948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:07:09.627240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58286 (local address 127.3.214.195:41497)
0924 16:07:09.627523 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0924 16:07:09.627530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:07:09.627647 (+ 117us) server_negotiation.cc:407] Connection header received
0924 16:07:09.627847 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:07:09.627858 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:07:09.627978 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:07:09.628124 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:07:09.629587 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.630794 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.632182 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:07:09.632862 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:07:09.633017 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:07:09.638681 (+ 5664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:07:09.638700 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:07:09.638726 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:07:09.638793 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:07:09.643215 (+ 4422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:07:09.644118 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:07:09.644130 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:07:09.644134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:07:09.644367 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:07:09.644783 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:07:09.644789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:07:09.644792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:07:09.645329 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:07:09.645700 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0924 16:07:09.646042 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 16:07:09.646512 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260924 16:07:09.648087 14927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:07:09.780457 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 14803
I20260924 16:07:09.801290 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 14438
I20260924 16:07:09.802832 14500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:07:09.931983 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 14438
2026-09-24T16:07:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4686 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:07:10 dist-test-slave-1gc3 krb5kdc[14984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:07:10 dist-test-slave-1gc3 krb5kdc[14984](info): set up 2 sockets
Sep 24 16:07:10 dist-test-slave-1gc3 krb5kdc[14984](info): commencing operation
krb5kdc: starting...
W20260924 16:07:11.005738 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.006s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:07:11 dist-test-slave-1gc3 krb5kdc[14984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:07:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:07:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.214.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.214.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.214.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/rangeradmin_127.3.214.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.214.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/rangeradmin_127.3.214.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.214.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.214.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.214.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/rangerlookup_127.3.214.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.214.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/rangerlookup_127.3.214.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.214.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.214.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.214.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/HTTP_127.3.214.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.214.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/HTTP_127.3.214.212@KRBTEST.COM.keytab.
I20260924 16:07:11.410737 3931 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHlT5a5/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/postgres -l logfile start
2026-09-24 16:07:14.187 UTC [15014] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 16:07:14.187 UTC [15014] LOG: listening on IPv4 address "127.3.214.212", port 49847
2026-09-24 16:07:14.192 UTC [15014] LOG: listening on Unix socket "/tmp/.s.PGSQL.49847"
2026-09-24 16:07:14.219 UTC [15017] LOG: database system was shut down at 2026-09-24 16:07:12 UTC
2026-09-24 16:07:14.224 UTC [15014] LOG: database system is ready to accept connections
I20260924 16:07:15.136003 3931 mini_postgres.cc:96] Postgres bound to 49847
2026-09-24 16:07:15.160 UTC [15025] FATAL: database "slave" does not exist
127.3.214.212:49847 - accepting connections
I20260924 16:07:15.161134 3931 mini_ranger.cc:161] Starting Ranger...
I20260924 16:07:15.196398 3931 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 16:07:15.743113 3931 mini_ranger.cc:87] Created ranger Postgres database
I20260924 16:07:15.743316 3931 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ranger-admin
2026-09-24 16:07:16,092 [I] DB FLAVOR :POSTGRES
2026-09-24 16:07:16,093 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 16:07:16,093 [I] Checking connection..
2026-09-24 16:07:16,093 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:49847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 16:07:16,378 [I] Checking connection passed.
2026-09-24 16:07:16,379 [I] --------- Verifying version history table ---------
2026-09-24 16:07:16,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:49847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:07:16,678 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 16:07:16,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:49847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:07:16,950 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 16:07:16,950 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 16:07:16,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:49847/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 16:07:17.225 UTC [15090] WARNING: there is no transaction in progress
2026-09-24 16:07:17,233 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 16:07:17,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:49847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:07:17,494 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 16:07:17,494 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 16:07:17,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:49847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 16:07:17,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:49847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 16:07:18,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:49847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1gc3.c.gcp-upstream.internal','N') ;"
2026-09-24 16:07:18,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:49847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 16:07:18,598 [I] Table x_portal_user does not exist in database ranger
2026-09-24 16:07:18,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:49847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 16:07:18,879 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 16:07:18,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:49847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 16:07:19,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:49847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 16:07:19,453 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 16:07:19,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:49847/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 16:07:20.605 UTC [15198] WARNING: there is no transaction in progress
2026-09-24 16:07:20.636 UTC [15198] WARNING: there is no transaction in progress
2026-09-24 16:07:20.662 UTC [15198] WARNING: there is no transaction in progress
2026-09-24 16:07:20.681 UTC [15198] WARNING: there is no transaction in progress
2026-09-24 16:07:20.703 UTC [15198] WARNING: there is no transaction in progress
2026-09-24 16:07:20.727 UTC [15198] WARNING: there is no transaction in progress
2026-09-24 16:07:20.792 UTC [15198] WARNING: there is no transaction in progress
2026-09-24 16:07:20.800 UTC [15198] WARNING: there is no transaction in progress
2026-09-24 16:07:20.817 UTC [15198] WARNING: there is no transaction in progress
2026-09-24 16:07:20.828 UTC [15198] WARNING: there is no transaction in progress
2026-09-24 16:07:20.841 UTC [15198] WARNING: there is no transaction in progress
2026-09-24 16:07:20.858 UTC [15198] WARNING: there is no transaction in progress
2026-09-24 16:07:20.871 UTC [15198] WARNING: there is no transaction in progress
2026-09-24 16:07:20.879 UTC [15198] WARNING: there is no transaction in progress
2026-09-24 16:07:20.888 UTC [15198] WARNING: there is no transaction in progress
2026-09-24 16:07:20.896 UTC [15198] WARNING: there is no transaction in progress
2026-09-24 16:07:21,732 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 16:07:21,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:49847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 16:07:22,000 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 16:07:22,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:49847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 16:07:22,275 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 16:07:22,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:49847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 16:07:22,552 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 16:07:22,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:49847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 16:07:22,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:49847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 16:07:23,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:49847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1gc3.c.gcp-upstream.internal';"
2026-09-24 16:07:23,410 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 16:07:23,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:49847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 16:07:23,708 [I] DB_PATCHES have already been applied
I20260924 16:07:23.715163 3931 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHlT5a5/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 16:07:23.715270 3931 mini_ranger.cc:193] Using host: 127.3.214.212
I20260924 16:07:23.735809 3931 mini_ranger.cc:227] Ranger admin URL: http://127.3.214.212:55321
Sep 24, 2026 4:07:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:07:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:07:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 4:07:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 4:07:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 4:07:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 4:07:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 16:07:25.926005 3931 mini_ranger.cc:160] Time spent starting Ranger: real 10.765s user 0.004s sys 0.067s
Sep 24, 2026 4:07:48 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 4:07:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 4:07:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 4:07:49 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 4:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 4:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 4:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 4:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 4:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 4:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 4:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 4:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 4:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 4:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 4:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 4:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 4:07:49 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 4:07:50 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 16:07:50.824007 3931 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.214.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.214.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.214.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/rangerkms_127.3.214.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.214.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/rangerkms_127.3.214.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.214.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.214.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.214.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/HTTP_127.3.214.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.214.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/HTTP_127.3.214.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 16:07:51 dist-test-slave-1gc3 krb5kdc[14984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266071, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 16:07:51.035209 3931 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 16:07:51.070686 3931 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 16:07:52.080235 3931 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 16:07:52.080425 3931 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ranger-kms
I20260924 16:07:52.080576 3931 mini_ranger_kms.cc:227] Using postgres at 127.3.214.212:49847
2026-09-24 16:07:53,027 [I] DB FLAVOR :POSTGRES
2026-09-24 16:07:53,027 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 16:07:53,027 [I] Checking connection
2026-09-24 16:07:53,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:49847/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 16:07:53,296 [I] connection success
2026-09-24 16:07:53,297 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 16:07:53,297 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 16:07:53,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:49847/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 16:07:53,573 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 16:07:53,574 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 16:07:53,574 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 16:07:53,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:49847/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 16:07:53,885 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 16:07:54.412194 3931 mini_ranger_kms.cc:310] Created kms service
I20260924 16:07:54.607237 3931 mini_ranger_kms.cc:326] Created kudu user
I20260924 16:07:54.664994 3931 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 16:07:54 dist-test-slave-1gc3 krb5kdc[14984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266071, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
I20260924 16:07:55.137888 3931 mini_ranger_kms.cc:384] Added ranger policy
I20260924 16:07:55.139413 3931 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ranger-kms:/tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 16:07:55.139477 3931 mini_ranger_kms.cc:240] Using host: 127.3.214.212
I20260924 16:07:55.166241 3931 mini_ranger_kms.cc:276] Ranger KMS PID: 15381
I20260924 16:07:55.166502 3931 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.214.212:52727
Sep 24, 2026 4:07:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:07:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:07:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 4:07:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 4:07:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 4:07:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 16:07:57.338512 3931 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.303s user 0.169s sys 0.469s
I20260924 16:07:57.338791 3931 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 16:07:57.339051 3931 mini_ranger_kms.cc:401] 127.3.214.212:52727/kms/v1/keys
Sep 24 16:07:57 dist-test-slave-1gc3 krb5kdc[14984](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790266077, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.214.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 16:07:58 dist-test-slave-1gc3 krb5kdc[14984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266077, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.214.212@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
Sep 24 16:07:58 dist-test-slave-1gc3 krb5kdc[14984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266077, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.214.212@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
Sep 24 16:08:01 dist-test-slave-1gc3 krb5kdc[14984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:08:01.451848 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:36739
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:36739
--ranger_config_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.214.212:52727/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:08:01.842916 15432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:08:01.843264 15432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:08:01.843411 15432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:08:01.851712 15432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:08:01.851855 15432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:08:01.851925 15432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:08:01.852018 15432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:08:01.852082 15432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:08:01.864254 15432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.214.212:52727/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:36739
--ranger_config_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:36739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:08:01.865744 15432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:08:01.867241 15432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:08:01.884173 15438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:08:01.884961 15440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:08:01.885013 15437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:08:01.888129 15432 server_base.cc:1034] running on GCE node
I20260924 16:08:01.889358 15432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:08:01.891746 15432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:08:01.893052 15432 hybrid_clock.cc:648] HybridClock initialized: now 1790266081892986 us; error 61 us; skew 500 ppm
I20260924 16:08:01.893525 15432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:08:01 dist-test-slave-1gc3 krb5kdc[14984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266081, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:08:01.897744 15432 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:08:01.900040 15432 webserver.cc:460] Webserver started at http://127.3.214.254:41839/ using document root <none> and password file <none>
I20260924 16:08:01.900951 15432 fs_manager.cc:359] Metadata directory not provided
I20260924 16:08:01.901091 15432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:08:01.901449 15432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:08:01 dist-test-slave-1gc3 krb5kdc[14984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266081, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
I20260924 16:08:01.986017 15432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "627ee5b3d4524c6f82e65bfe2f920822"
format_stamp: "Formatted at 2026-09-24 16:08:01 on dist-test-slave-1gc3"
server_key: "946be72ebc51ec5d975eecaab438faab"
server_key_iv: "741624aa3bec2642218916783f43ed56"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:01.987151 15432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "627ee5b3d4524c6f82e65bfe2f920822"
format_stamp: "Formatted at 2026-09-24 16:08:01 on dist-test-slave-1gc3"
server_key: "946be72ebc51ec5d975eecaab438faab"
server_key_iv: "741624aa3bec2642218916783f43ed56"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:01.994005 15432 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260924 16:08:02.020893 15447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:08:02.023267 15432 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260924 16:08:02.023571 15432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "627ee5b3d4524c6f82e65bfe2f920822"
format_stamp: "Formatted at 2026-09-24 16:08:01 on dist-test-slave-1gc3"
server_key: "946be72ebc51ec5d975eecaab438faab"
server_key_iv: "741624aa3bec2642218916783f43ed56"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:02.023872 15432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:08:02.045410 15432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:08:02.050978 15432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:08:02.051333 15432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:08:02.074652 15432 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:36739
I20260924 16:08:02.074692 15498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:36739 every 8 connection(s)
I20260924 16:08:02.076499 15432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:08:02.077950 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 15432
I20260924 16:08:02.078217 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:08:02.079082 3931 external_mini_cluster.cc:1428] Setting key be41cd04967bc677bd74c6809e12d081
I20260924 16:08:02.082211 15499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 16:08:02 dist-test-slave-1gc3 krb5kdc[14984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:08:02.097105 15499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822: Bootstrap starting.
I20260924 16:08:02.101449 15499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822: Neither blocks nor log segments found. Creating new log.
I20260924 16:08:02.102896 15499 log.cc:824] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822: Log is configured to *not* fsync() on all Append() calls
I20260924 16:08:02.107288 15499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822: No bootstrap required, opened a new log
I20260924 16:08:02.108117 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:02.081649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37846 (local address 127.3.214.254:36739)
0924 16:08:02.082718 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0924 16:08:02.082734 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:08:02.082791 (+ 57us) server_negotiation.cc:407] Connection header received
0924 16:08:02.084847 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:08:02.084877 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:08:02.085428 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:08:02.085899 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:08:02.087910 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:02.089708 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:02.091692 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:02.092587 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:02.092769 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:02.098344 (+ 5575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:08:02.098359 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:08:02.098382 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:08:02.098449 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:08:02.102521 (+ 4072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:08:02.103646 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:08:02.103661 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:08:02.103671 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:08:02.103902 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:08:02.104478 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:08:02.104483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:08:02.104486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:08:02.105350 (+ 864us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:08:02.105712 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0924 16:08:02.106246 (+ 534us) server_negotiation.cc:299] Negotiation successful
0924 16:08:02.106724 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":506,"threads_started":1}
I20260924 16:08:02.116251 15499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627ee5b3d4524c6f82e65bfe2f920822" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 36739 } }
I20260924 16:08:02.116905 15499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:08:02.117002 15499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 627ee5b3d4524c6f82e65bfe2f920822, State: Initialized, Role: FOLLOWER
I20260924 16:08:02.117683 15499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627ee5b3d4524c6f82e65bfe2f920822" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 36739 } }
I20260924 16:08:02.117939 15499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:08:02.118144 15499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:08:02.118376 15499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:08:02.121003 15499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627ee5b3d4524c6f82e65bfe2f920822" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 36739 } }
I20260924 16:08:02.121657 15499 leader_election.cc:302] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 627ee5b3d4524c6f82e65bfe2f920822; no voters:
I20260924 16:08:02.122565 15499 leader_election.cc:288] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:08:02.122766 15504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:08:02.123922 15504 raft_consensus.cc:686] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822 [term 1 LEADER]: Becoming Leader. State: Replica: 627ee5b3d4524c6f82e65bfe2f920822, State: Running, Role: LEADER
I20260924 16:08:02.124759 15504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627ee5b3d4524c6f82e65bfe2f920822" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 36739 } }
I20260924 16:08:02.125624 15499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:08:02.128224 15505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "627ee5b3d4524c6f82e65bfe2f920822" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627ee5b3d4524c6f82e65bfe2f920822" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 36739 } } }
I20260924 16:08:02.128983 15505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822 [sys.catalog]: This master's current role is: LEADER
I20260924 16:08:02.129617 15506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 627ee5b3d4524c6f82e65bfe2f920822. Latest consensus state: current_term: 1 leader_uuid: "627ee5b3d4524c6f82e65bfe2f920822" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627ee5b3d4524c6f82e65bfe2f920822" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 36739 } } }
I20260924 16:08:02.130339 15506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822 [sys.catalog]: This master's current role is: LEADER
I20260924 16:08:02.131686 15510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:08:02.139199 15510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:08:02.142540 15499 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 16:08:02.164395 15510 catalog_manager.cc:1347] Generated new cluster ID: 9c9bf6b4c22a4969af772428c0eb90ff
I20260924 16:08:02.164621 15510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:08:02.180461 15510 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:08:02.182931 15510 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:08:02.194284 15510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822: Generated new TSK 0
I20260924 16:08:02.196053 15510 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.214.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
I20260924 16:08:03.223279 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.214.212:52727/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:36739
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:08:03.747463 15544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:08:03.748052 15544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:08:03.748260 15544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:08:03.760776 15544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:08:03.760985 15544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:08:03.761219 15544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
Sep 24 16:08:03 dist-test-slave-1gc3 krb5kdc[14984](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790266083, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:08:03.778716 15544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.214.212:52727/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:36739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:08:03.781134 15544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:08:03.783653 15544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:08:03.809355 15554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:08:03.811391 15553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:08:03.815914 15556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:08:03.817037 15544 server_base.cc:1034] running on GCE node
I20260924 16:08:03.817984 15544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:08:03.820951 15544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:08:03.822271 15544 hybrid_clock.cc:648] HybridClock initialized: now 1790266083822158 us; error 113 us; skew 500 ppm
I20260924 16:08:03.822690 15544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:08:03 dist-test-slave-1gc3 krb5kdc[14984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266083, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:08:03.826584 15544 init.cc:377] Logged in from keytab as kudu/127.3.214.193@KRBTEST.COM (short username kudu)
I20260924 16:08:03.831754 15544 webserver.cc:460] Webserver started at http://127.3.214.193:44259/ using document root <none> and password file <none>
I20260924 16:08:03.832647 15544 fs_manager.cc:359] Metadata directory not provided
I20260924 16:08:03.832791 15544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:08:03.833158 15544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:08:03 dist-test-slave-1gc3 krb5kdc[14984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266083, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
I20260924 16:08:03.858954 15544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "8c14c4cae2e84b44ba5bb11fdabbc4ee"
format_stamp: "Formatted at 2026-09-24 16:08:03 on dist-test-slave-1gc3"
server_key: "582dc5b340b65bef96c43d71ab513844"
server_key_iv: "fb13750bd5fcb55f15a2164d44f7417e"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:03.860561 15544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "8c14c4cae2e84b44ba5bb11fdabbc4ee"
format_stamp: "Formatted at 2026-09-24 16:08:03 on dist-test-slave-1gc3"
server_key: "582dc5b340b65bef96c43d71ab513844"
server_key_iv: "fb13750bd5fcb55f15a2164d44f7417e"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:03.872617 15544 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.007s sys 0.000s
I20260924 16:08:03.902699 15564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:08:03.905270 15544 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.005s
I20260924 16:08:03.905547 15544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "8c14c4cae2e84b44ba5bb11fdabbc4ee"
format_stamp: "Formatted at 2026-09-24 16:08:03 on dist-test-slave-1gc3"
server_key: "582dc5b340b65bef96c43d71ab513844"
server_key_iv: "fb13750bd5fcb55f15a2164d44f7417e"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:03.905843 15544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:08:03.926390 15544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:08:03.932257 15544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:08:03.932668 15544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:08:03.934305 15544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:08:03.936751 15544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:08:03.936879 15544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:08:03.937053 15544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:08:03.937134 15544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:08:03.985620 15544 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:45439
I20260924 16:08:03.985621 15676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:45439 every 8 connection(s)
I20260924 16:08:03.988145 15544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:08:03.997922 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 15544
I20260924 16:08:03.998231 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:08:03.999243 3931 external_mini_cluster.cc:1428] Setting key 7207ef996a9c71c5bcee175b817b126e
Sep 24 16:08:04 dist-test-slave-1gc3 krb5kdc[14984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266083, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:08:04.018000 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:03.991967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:33475 (local address 127.3.214.254:36739)
0924 16:08:03.992661 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0924 16:08:03.992668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:08:03.994604 (+ 1936us) server_negotiation.cc:407] Connection header received
0924 16:08:03.997359 (+ 2755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:08:03.997369 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:08:03.997540 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:08:03.997754 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:08:04.000312 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:04.001889 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:04.004883 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:04.005423 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:04.005568 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:04.009810 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:08:04.009816 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:08:04.009821 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:08:04.009865 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:08:04.013374 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:08:04.014711 (+ 1337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:08:04.014718 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:08:04.014720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:08:04.014905 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:08:04.015526 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:08:04.015532 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:08:04.015536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:08:04.016026 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:08:04.016260 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 16:08:04.017205 (+ 945us) server_negotiation.cc:299] Negotiation successful
0924 16:08:04.017523 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":356,"threads_started":1}
I20260924 16:08:04.019145 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:03.992636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:36739 (local address 127.3.214.193:33475)
0924 16:08:03.993776 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0924 16:08:03.993854 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0924 16:08:03.996789 (+ 2935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:08:03.998165 (+ 1376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:08:03.998182 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:08:03.998850 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:08:03.999970 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:08:04.000004 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:08:04.002259 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:08:04.002266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:08:04.004615 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:08:04.004648 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:08:04.005743 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:08:04.005754 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:08:04.006032 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:04.006946 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:08:04.006978 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:08:04.009451 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:08:04.013660 (+ 4209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:08:04.013673 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:08:04.013692 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:08:04.014524 (+ 832us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:08:04.015040 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:08:04.015046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:08:04.015050 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:08:04.015352 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:08:04.016198 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:08:04.016215 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:08:04.016698 (+ 483us) client_negotiation.cc:769] Sending connection context
0924 16:08:04.017029 (+ 331us) client_negotiation.cc:240] Negotiation successful
0924 16:08:04.017686 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":381,"threads_started":1}
I20260924 16:08:04.021457 15677 heartbeater.cc:347] Connected to a master server at 127.3.214.254:36739
I20260924 16:08:04.021908 15677 heartbeater.cc:464] Registering TS with master...
I20260924 16:08:04.022934 15677 heartbeater.cc:511] Master 127.3.214.254:36739 requested a full tablet report, sending...
I20260924 16:08:04.027756 15464 ts_manager.cc:194] Registered new tserver with Master: 8c14c4cae2e84b44ba5bb11fdabbc4ee (127.3.214.193:45439)
I20260924 16:08:04.030340 15464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:33475
WARNING: no policy specified for kudu/127.3.214.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
I20260924 16:08:04.074263 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.214.212:52727/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:36739
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:08:04 dist-test-slave-1gc3 krb5kdc[14984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266083, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
W20260924 16:08:04.582638 15683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:08:04.583060 15683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:08:04.583243 15683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:08:04.593221 15683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:08:04.593508 15683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:08:04.593752 15683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:08:04.607873 15683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.214.212:52727/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:36739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:08:04.609908 15683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:08:04.611817 15683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:08:04.631477 15688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:08:04.633049 15689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:08:04.641907 15683 server_base.cc:1034] running on GCE node
W20260924 16:08:04.642168 15691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:08:04.643074 15683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:08:04.644280 15683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:08:04.645574 15683 hybrid_clock.cc:648] HybridClock initialized: now 1790266084645505 us; error 66 us; skew 500 ppm
I20260924 16:08:04.645954 15683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:08:04 dist-test-slave-1gc3 krb5kdc[14984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266084, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:08:04.649464 15683 init.cc:377] Logged in from keytab as kudu/127.3.214.194@KRBTEST.COM (short username kudu)
I20260924 16:08:04.658531 15683 webserver.cc:460] Webserver started at http://127.3.214.194:37387/ using document root <none> and password file <none>
I20260924 16:08:04.659440 15683 fs_manager.cc:359] Metadata directory not provided
I20260924 16:08:04.659586 15683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:08:04.659948 15683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:08:04 dist-test-slave-1gc3 krb5kdc[14984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266084, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
I20260924 16:08:04.686033 15683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "e72b35e925cd485181a5e2ca83f4c3ad"
format_stamp: "Formatted at 2026-09-24 16:08:04 on dist-test-slave-1gc3"
server_key: "9d759001df42e268407f75ede8ba350d"
server_key_iv: "439c7e7f4fd5b488f0248e8fe3b28215"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:04.687471 15683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "e72b35e925cd485181a5e2ca83f4c3ad"
format_stamp: "Formatted at 2026-09-24 16:08:04 on dist-test-slave-1gc3"
server_key: "9d759001df42e268407f75ede8ba350d"
server_key_iv: "439c7e7f4fd5b488f0248e8fe3b28215"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:04.709235 15683 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.007s sys 0.012s
I20260924 16:08:04.743359 15698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:08:04.746074 15683 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.002s sys 0.008s
I20260924 16:08:04.746511 15683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "e72b35e925cd485181a5e2ca83f4c3ad"
format_stamp: "Formatted at 2026-09-24 16:08:04 on dist-test-slave-1gc3"
server_key: "9d759001df42e268407f75ede8ba350d"
server_key_iv: "439c7e7f4fd5b488f0248e8fe3b28215"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:04.746959 15683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:08:04.791965 15683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:08:04.798686 15683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:08:04.799222 15683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:08:04.811129 15683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:08:04.823375 15683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:08:04.823683 15683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 16:08:04.823915 15683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:08:04.824045 15683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:08:04 dist-test-slave-1gc3 krb5kdc[14984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266083, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
I20260924 16:08:05.035712 15677 heartbeater.cc:503] Master 127.3.214.254:36739 was elected leader, sending a full tablet report...
I20260924 16:08:05.331480 15683 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:37779
I20260924 16:08:05.331501 15812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:37779 every 8 connection(s)
I20260924 16:08:05.333431 15683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:08:05.337096 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 15683
I20260924 16:08:05.337298 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:08:05.338251 3931 external_mini_cluster.cc:1428] Setting key b75fba2bf568c8426a555fc7c2901f27
Sep 24 16:08:05 dist-test-slave-1gc3 krb5kdc[14984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266084, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:08:05.376261 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:05.338828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:34689 (local address 127.3.214.254:36739)
0924 16:08:05.349818 (+ 10990us) server_negotiation.cc:206] Beginning negotiation
0924 16:08:05.349825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:08:05.354153 (+ 4328us) server_negotiation.cc:407] Connection header received
0924 16:08:05.355795 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:08:05.355801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:08:05.355907 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:08:05.355991 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:08:05.358713 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:05.359770 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:05.362035 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:05.362643 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:05.362749 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:05.368073 (+ 5324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:08:05.368080 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:08:05.368084 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:08:05.368136 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:08:05.371728 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:08:05.372883 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:08:05.372889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:08:05.372893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:08:05.373088 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:08:05.373696 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:08:05.373701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:08:05.373704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:08:05.374137 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:08:05.374301 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 16:08:05.375533 (+ 1232us) server_negotiation.cc:299] Negotiation successful
0924 16:08:05.375820 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10826,"thread_start_us":10675,"threads_started":1}
I20260924 16:08:05.379334 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:05.338082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:36739 (local address 127.3.214.194:34689)
0924 16:08:05.351448 (+ 13366us) negotiation.cc:107] Waiting for socket to connect
0924 16:08:05.351520 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 16:08:05.355441 (+ 3921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:08:05.356695 (+ 1254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:08:05.356708 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:08:05.357446 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:08:05.358499 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:08:05.358528 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:08:05.359977 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:08:05.359982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:08:05.361856 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:08:05.361873 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:08:05.363174 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:08:05.363179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:08:05.363501 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:05.364438 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:08:05.364482 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:08:05.367731 (+ 3249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:08:05.372000 (+ 4269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:08:05.372012 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:08:05.372050 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:08:05.372732 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:08:05.373251 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:08:05.373256 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:08:05.373259 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:08:05.373570 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:08:05.374405 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:08:05.374415 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:08:05.375007 (+ 592us) client_negotiation.cc:769] Sending connection context
0924 16:08:05.376871 (+ 1864us) client_negotiation.cc:240] Negotiation successful
0924 16:08:05.377570 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12895,"thread_start_us":15916,"threads_started":1}
I20260924 16:08:05.381592 15813 heartbeater.cc:347] Connected to a master server at 127.3.214.254:36739
I20260924 16:08:05.381971 15813 heartbeater.cc:464] Registering TS with master...
I20260924 16:08:05.383132 15813 heartbeater.cc:511] Master 127.3.214.254:36739 requested a full tablet report, sending...
I20260924 16:08:05.385200 15464 ts_manager.cc:194] Registered new tserver with Master: e72b35e925cd485181a5e2ca83f4c3ad (127.3.214.194:37779)
I20260924 16:08:05.386457 15464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:34689
WARNING: no policy specified for kudu/127.3.214.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
I20260924 16:08:05.434516 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.214.212:52727/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:36739
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:08:05.879739 15826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:08:05.880164 15826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:08:05.880314 15826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:08:05.888957 15826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:08:05.889137 15826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:08:05.889310 15826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:08:05.901157 15826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.214.212:52727/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:36739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:08:05.903093 15826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:08:05.904790 15826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:08:05.922289 15832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:08:05.922360 15831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:08:05.922226 15834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:08:05.927731 15826 server_base.cc:1034] running on GCE node
I20260924 16:08:05.928807 15826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:08:05.930325 15826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:08:05.931625 15826 hybrid_clock.cc:648] HybridClock initialized: now 1790266085931551 us; error 55 us; skew 500 ppm
I20260924 16:08:05.932106 15826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:08:05 dist-test-slave-1gc3 krb5kdc[14984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266085, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:08:05.936625 15826 init.cc:377] Logged in from keytab as kudu/127.3.214.195@KRBTEST.COM (short username kudu)
I20260924 16:08:05.939178 15826 webserver.cc:460] Webserver started at http://127.3.214.195:40477/ using document root <none> and password file <none>
I20260924 16:08:05.940197 15826 fs_manager.cc:359] Metadata directory not provided
I20260924 16:08:05.940346 15826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:08:05.940747 15826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:08:05 dist-test-slave-1gc3 krb5kdc[14984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266085, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
I20260924 16:08:05.959355 15826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "1927c9f32f1646cca00301db59382ceb"
format_stamp: "Formatted at 2026-09-24 16:08:05 on dist-test-slave-1gc3"
server_key: "b8fc63f966d71610e72a529c825d1b60"
server_key_iv: "df5b3329eb3ccede6642542b3dffc8b3"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:05.960331 15826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "1927c9f32f1646cca00301db59382ceb"
format_stamp: "Formatted at 2026-09-24 16:08:05 on dist-test-slave-1gc3"
server_key: "b8fc63f966d71610e72a529c825d1b60"
server_key_iv: "df5b3329eb3ccede6642542b3dffc8b3"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:05.967366 15826 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260924 16:08:05.986029 15841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:08:05.988623 15826 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260924 16:08:05.988912 15826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "1927c9f32f1646cca00301db59382ceb"
format_stamp: "Formatted at 2026-09-24 16:08:05 on dist-test-slave-1gc3"
server_key: "b8fc63f966d71610e72a529c825d1b60"
server_key_iv: "df5b3329eb3ccede6642542b3dffc8b3"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:05.989218 15826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:08:06.027096 15826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:08:06.032161 15826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:08:06.032523 15826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:08:06.033680 15826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:08:06.035632 15826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:08:06.035734 15826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:08:06.035873 15826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:08:06.035934 15826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:08:06.075259 15826 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:36921
I20260924 16:08:06.075321 15953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:36921 every 8 connection(s)
I20260924 16:08:06.077030 15826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:08:06.083786 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 15826
I20260924 16:08:06.084048 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:08:06.084681 3931 external_mini_cluster.cc:1428] Setting key 92d649d34cfd3c3acd0078b6a877314a
Sep 24 16:08:06 dist-test-slave-1gc3 krb5kdc[14984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266085, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:08:06.108278 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:06.080588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:50227 (local address 127.3.214.254:36739)
0924 16:08:06.081198 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0924 16:08:06.081203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:08:06.082431 (+ 1228us) server_negotiation.cc:407] Connection header received
0924 16:08:06.085360 (+ 2929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:08:06.085369 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:08:06.085487 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:08:06.085658 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:08:06.088068 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.089090 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.091070 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.091553 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.091682 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:06.096522 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:08:06.096537 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:08:06.096543 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:08:06.096599 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:08:06.103946 (+ 7347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:08:06.105177 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:08:06.105183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:08:06.105185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:08:06.105400 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:08:06.106164 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:08:06.106206 (+ 42us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:08:06.106208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:08:06.106576 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:08:06.106738 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 16:08:06.107731 (+ 993us) server_negotiation.cc:299] Negotiation successful
0924 16:08:06.107963 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":327,"threads_started":1}
I20260924 16:08:06.109583 15957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:06.081135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:36739 (local address 127.3.214.195:50227)
0924 16:08:06.082216 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0924 16:08:06.082275 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:08:06.084888 (+ 2613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:08:06.086050 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:08:06.086064 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:08:06.086882 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:08:06.087849 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:08:06.087873 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.089276 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.089281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:08:06.090866 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:08:06.090884 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.091888 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.091895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:08:06.092184 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:06.093133 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:08:06.093189 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:08:06.096026 (+ 2837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:08:06.104196 (+ 8170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:08:06.104210 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:08:06.104236 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:08:06.104961 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:08:06.105608 (+ 647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:08:06.105613 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:08:06.105615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:08:06.105936 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:08:06.106716 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:08:06.106725 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:08:06.107247 (+ 522us) client_negotiation.cc:769] Sending connection context
0924 16:08:06.107591 (+ 344us) client_negotiation.cc:240] Negotiation successful
0924 16:08:06.108162 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":400,"threads_started":1}
I20260924 16:08:06.111613 15954 heartbeater.cc:347] Connected to a master server at 127.3.214.254:36739
I20260924 16:08:06.112025 15954 heartbeater.cc:464] Registering TS with master...
I20260924 16:08:06.113085 15954 heartbeater.cc:511] Master 127.3.214.254:36739 requested a full tablet report, sending...
I20260924 16:08:06.115383 15464 ts_manager.cc:194] Registered new tserver with Master: 1927c9f32f1646cca00301db59382ceb (127.3.214.195:36921)
I20260924 16:08:06.116416 15464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:50227
I20260924 16:08:06.129180 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:08:06.150775 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:06.135390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48478 (local address 127.3.214.254:36739)
0924 16:08:06.135713 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0924 16:08:06.135721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:08:06.136126 (+ 405us) server_negotiation.cc:407] Connection header received
0924 16:08:06.136580 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:08:06.136587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:08:06.136788 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:08:06.136969 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:08:06.138690 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.139548 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.141179 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.141675 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.141806 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:06.144607 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:08:06.144614 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:08:06.144618 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:08:06.144668 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:08:06.147715 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:08:06.148585 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:08:06.148590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:08:06.148593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:08:06.148763 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:08:06.149148 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:08:06.149151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:08:06.149153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:08:06.149516 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:08:06.149731 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 16:08:06.150175 (+ 444us) server_negotiation.cc:299] Negotiation successful
0924 16:08:06.150454 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260924 16:08:06.156468 15464 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:08:06.165205 15464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:08:06.208492 15968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:06.182277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59062 (local address 127.3.214.195:36921)
0924 16:08:06.195685 (+ 13408us) server_negotiation.cc:206] Beginning negotiation
0924 16:08:06.195699 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:08:06.197678 (+ 1979us) server_negotiation.cc:407] Connection header received
0924 16:08:06.198282 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:08:06.198294 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:08:06.198647 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:08:06.198855 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:08:06.200196 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.202089 (+ 1893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.205645 (+ 3556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.207085 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.207223 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:06.207545 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0924 16:08:06.207849 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 16:08:06.208065 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13150,"thread_start_us":12971,"threads_started":1}
I20260924 16:08:06.209909 15970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:06.182092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:36921 (local address 127.0.0.1:59062)
0924 16:08:06.197415 (+ 15323us) negotiation.cc:107] Waiting for socket to connect
0924 16:08:06.197493 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0924 16:08:06.197918 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:08:06.199300 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:08:06.199309 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:08:06.199350 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:08:06.199919 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:08:06.199933 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.202379 (+ 2446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.202387 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:08:06.205333 (+ 2946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:08:06.205351 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.207316 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.207323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:08:06.207571 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:06.207627 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 16:08:06.207804 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 16:08:06.209428 (+ 1624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14888,"mutex_wait_us":1325,"thread_start_us":21178,"threads_started":1}
I20260924 16:08:06.212666 15972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:06.183930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37444 (local address 127.3.214.194:37779)
0924 16:08:06.202696 (+ 18766us) server_negotiation.cc:206] Beginning negotiation
0924 16:08:06.202710 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:08:06.204359 (+ 1649us) server_negotiation.cc:407] Connection header received
0924 16:08:06.204473 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:08:06.204479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:08:06.204686 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:08:06.204880 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:08:06.205707 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.207302 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.210041 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.211369 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.211518 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:06.211767 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 16:08:06.212123 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 16:08:06.212322 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18509,"thread_start_us":18172,"threads_started":1}
I20260924 16:08:06.213802 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:06.182678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:37779 (local address 127.0.0.1:37444)
0924 16:08:06.204057 (+ 21379us) negotiation.cc:107] Waiting for socket to connect
0924 16:08:06.204074 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:08:06.204199 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:08:06.204970 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:08:06.204976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:08:06.205011 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:08:06.205455 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:08:06.205467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.207510 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.207516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:08:06.209786 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:08:06.209806 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.211720 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.211729 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:08:06.211927 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:06.211955 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 16:08:06.212046 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 16:08:06.212220 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21243,"mutex_wait_us":2,"thread_start_us":14154,"threads_started":1}
I20260924 16:08:06.214557 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:06.182272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:45439 (local address 127.0.0.1:33598)
0924 16:08:06.203618 (+ 21346us) negotiation.cc:107] Waiting for socket to connect
0924 16:08:06.203645 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 16:08:06.203863 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:08:06.205540 (+ 1677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:08:06.205545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:08:06.205573 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:08:06.206027 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:08:06.206040 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.208731 (+ 2691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.208738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:08:06.211070 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:08:06.211088 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.213557 (+ 2469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.213565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:08:06.213754 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:06.213801 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 16:08:06.213894 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 16:08:06.214056 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21212,"mutex_wait_us":48,"thread_start_us":20790,"threads_started":1}
I20260924 16:08:06.214929 15971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:06.183435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33598 (local address 127.3.214.193:45439)
0924 16:08:06.196587 (+ 13152us) server_negotiation.cc:206] Beginning negotiation
0924 16:08:06.196598 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:08:06.203854 (+ 7256us) server_negotiation.cc:407] Connection header received
0924 16:08:06.204189 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:08:06.204201 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:08:06.204617 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:08:06.204881 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:08:06.206314 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.208505 (+ 2191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.211305 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.212952 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.213083 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:06.213369 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 16:08:06.214336 (+ 967us) server_negotiation.cc:299] Negotiation successful
0924 16:08:06.214569 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12885,"thread_start_us":12712,"threads_started":1}
I20260924 16:08:06.214967 15889 tablet_service.cc:1467] Processing CreateTablet for tablet 4845bab6fc0a4d5fbdf32666c046dc5b (DEFAULT_TABLE table=test-table [id=d664e5740fbe49cb8147b04d91710237]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:08:06.216804 15889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4845bab6fc0a4d5fbdf32666c046dc5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:08:06.218400 15748 tablet_service.cc:1467] Processing CreateTablet for tablet 4845bab6fc0a4d5fbdf32666c046dc5b (DEFAULT_TABLE table=test-table [id=d664e5740fbe49cb8147b04d91710237]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:08:06.219383 15612 tablet_service.cc:1467] Processing CreateTablet for tablet 4845bab6fc0a4d5fbdf32666c046dc5b (DEFAULT_TABLE table=test-table [id=d664e5740fbe49cb8147b04d91710237]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:08:06.220165 15748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4845bab6fc0a4d5fbdf32666c046dc5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:08:06.221169 15612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4845bab6fc0a4d5fbdf32666c046dc5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:08:06.234295 15974 tablet_bootstrap.cc:492] T 4845bab6fc0a4d5fbdf32666c046dc5b P e72b35e925cd485181a5e2ca83f4c3ad: Bootstrap starting.
I20260924 16:08:06.236886 15973 tablet_bootstrap.cc:492] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb: Bootstrap starting.
I20260924 16:08:06.238785 15975 tablet_bootstrap.cc:492] T 4845bab6fc0a4d5fbdf32666c046dc5b P 8c14c4cae2e84b44ba5bb11fdabbc4ee: Bootstrap starting.
I20260924 16:08:06.239074 15974 tablet_bootstrap.cc:654] T 4845bab6fc0a4d5fbdf32666c046dc5b P e72b35e925cd485181a5e2ca83f4c3ad: Neither blocks nor log segments found. Creating new log.
I20260924 16:08:06.240893 15973 tablet_bootstrap.cc:654] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb: Neither blocks nor log segments found. Creating new log.
I20260924 16:08:06.241861 15974 log.cc:824] T 4845bab6fc0a4d5fbdf32666c046dc5b P e72b35e925cd485181a5e2ca83f4c3ad: Log is configured to *not* fsync() on all Append() calls
I20260924 16:08:06.242313 15973 log.cc:824] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb: Log is configured to *not* fsync() on all Append() calls
I20260924 16:08:06.243439 15975 tablet_bootstrap.cc:654] T 4845bab6fc0a4d5fbdf32666c046dc5b P 8c14c4cae2e84b44ba5bb11fdabbc4ee: Neither blocks nor log segments found. Creating new log.
I20260924 16:08:06.245296 15975 log.cc:824] T 4845bab6fc0a4d5fbdf32666c046dc5b P 8c14c4cae2e84b44ba5bb11fdabbc4ee: Log is configured to *not* fsync() on all Append() calls
I20260924 16:08:06.246459 15974 tablet_bootstrap.cc:492] T 4845bab6fc0a4d5fbdf32666c046dc5b P e72b35e925cd485181a5e2ca83f4c3ad: No bootstrap required, opened a new log
I20260924 16:08:06.246861 15974 ts_tablet_manager.cc:1397] T 4845bab6fc0a4d5fbdf32666c046dc5b P e72b35e925cd485181a5e2ca83f4c3ad: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260924 16:08:06.247242 15973 tablet_bootstrap.cc:492] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb: No bootstrap required, opened a new log
I20260924 16:08:06.247529 15973 ts_tablet_manager.cc:1397] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260924 16:08:06.248464 15975 tablet_bootstrap.cc:492] T 4845bab6fc0a4d5fbdf32666c046dc5b P 8c14c4cae2e84b44ba5bb11fdabbc4ee: No bootstrap required, opened a new log
I20260924 16:08:06.248778 15975 ts_tablet_manager.cc:1397] T 4845bab6fc0a4d5fbdf32666c046dc5b P 8c14c4cae2e84b44ba5bb11fdabbc4ee: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 16:08:06.256832 15975 raft_consensus.cc:348] T 4845bab6fc0a4d5fbdf32666c046dc5b P 8c14c4cae2e84b44ba5bb11fdabbc4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927c9f32f1646cca00301db59382ceb" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 36921 } } peers { permanent_uuid: "8c14c4cae2e84b44ba5bb11fdabbc4ee" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 45439 } } peers { permanent_uuid: "e72b35e925cd485181a5e2ca83f4c3ad" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 37779 } }
I20260924 16:08:06.257457 15975 raft_consensus.cc:374] T 4845bab6fc0a4d5fbdf32666c046dc5b P 8c14c4cae2e84b44ba5bb11fdabbc4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:08:06.257576 15975 raft_consensus.cc:729] T 4845bab6fc0a4d5fbdf32666c046dc5b P 8c14c4cae2e84b44ba5bb11fdabbc4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c14c4cae2e84b44ba5bb11fdabbc4ee, State: Initialized, Role: FOLLOWER
I20260924 16:08:06.258319 15975 consensus_queue.cc:260] T 4845bab6fc0a4d5fbdf32666c046dc5b P 8c14c4cae2e84b44ba5bb11fdabbc4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927c9f32f1646cca00301db59382ceb" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 36921 } } peers { permanent_uuid: "8c14c4cae2e84b44ba5bb11fdabbc4ee" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 45439 } } peers { permanent_uuid: "e72b35e925cd485181a5e2ca83f4c3ad" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 37779 } }
I20260924 16:08:06.258633 15973 raft_consensus.cc:348] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927c9f32f1646cca00301db59382ceb" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 36921 } } peers { permanent_uuid: "8c14c4cae2e84b44ba5bb11fdabbc4ee" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 45439 } } peers { permanent_uuid: "e72b35e925cd485181a5e2ca83f4c3ad" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 37779 } }
I20260924 16:08:06.258708 15974 raft_consensus.cc:348] T 4845bab6fc0a4d5fbdf32666c046dc5b P e72b35e925cd485181a5e2ca83f4c3ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927c9f32f1646cca00301db59382ceb" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 36921 } } peers { permanent_uuid: "8c14c4cae2e84b44ba5bb11fdabbc4ee" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 45439 } } peers { permanent_uuid: "e72b35e925cd485181a5e2ca83f4c3ad" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 37779 } }
I20260924 16:08:06.259311 15973 raft_consensus.cc:374] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:08:06.259490 15973 raft_consensus.cc:729] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1927c9f32f1646cca00301db59382ceb, State: Initialized, Role: FOLLOWER
I20260924 16:08:06.260341 15973 consensus_queue.cc:260] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927c9f32f1646cca00301db59382ceb" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 36921 } } peers { permanent_uuid: "8c14c4cae2e84b44ba5bb11fdabbc4ee" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 45439 } } peers { permanent_uuid: "e72b35e925cd485181a5e2ca83f4c3ad" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 37779 } }
I20260924 16:08:06.261446 15975 ts_tablet_manager.cc:1428] T 4845bab6fc0a4d5fbdf32666c046dc5b P 8c14c4cae2e84b44ba5bb11fdabbc4ee: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260924 16:08:06.261736 15954 heartbeater.cc:503] Master 127.3.214.254:36739 was elected leader, sending a full tablet report...
I20260924 16:08:06.261963 15974 raft_consensus.cc:374] T 4845bab6fc0a4d5fbdf32666c046dc5b P e72b35e925cd485181a5e2ca83f4c3ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:08:06.262140 15974 raft_consensus.cc:729] T 4845bab6fc0a4d5fbdf32666c046dc5b P e72b35e925cd485181a5e2ca83f4c3ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72b35e925cd485181a5e2ca83f4c3ad, State: Initialized, Role: FOLLOWER
I20260924 16:08:06.262887 15974 consensus_queue.cc:260] T 4845bab6fc0a4d5fbdf32666c046dc5b P e72b35e925cd485181a5e2ca83f4c3ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927c9f32f1646cca00301db59382ceb" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 36921 } } peers { permanent_uuid: "8c14c4cae2e84b44ba5bb11fdabbc4ee" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 45439 } } peers { permanent_uuid: "e72b35e925cd485181a5e2ca83f4c3ad" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 37779 } }
I20260924 16:08:06.264436 15973 ts_tablet_manager.cc:1428] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb: Time spent starting tablet: real 0.017s user 0.012s sys 0.003s
I20260924 16:08:06.264964 15813 heartbeater.cc:503] Master 127.3.214.254:36739 was elected leader, sending a full tablet report...
I20260924 16:08:06.267666 15974 ts_tablet_manager.cc:1428] T 4845bab6fc0a4d5fbdf32666c046dc5b P e72b35e925cd485181a5e2ca83f4c3ad: Time spent starting tablet: real 0.020s user 0.016s sys 0.000s
I20260924 16:08:06.277791 15980 raft_consensus.cc:482] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:08:06.278364 15980 raft_consensus.cc:504] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927c9f32f1646cca00301db59382ceb" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 36921 } } peers { permanent_uuid: "8c14c4cae2e84b44ba5bb11fdabbc4ee" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 45439 } } peers { permanent_uuid: "e72b35e925cd485181a5e2ca83f4c3ad" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 37779 } }
I20260924 16:08:06.280601 15980 leader_election.cc:288] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c14c4cae2e84b44ba5bb11fdabbc4ee (127.3.214.193:45439), e72b35e925cd485181a5e2ca83f4c3ad (127.3.214.194:37779)
I20260924 16:08:06.288846 15971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:06.281154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:35425 (local address 127.3.214.193:45439)
0924 16:08:06.281383 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 16:08:06.281389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:08:06.281629 (+ 240us) server_negotiation.cc:407] Connection header received
0924 16:08:06.281871 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:08:06.281877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:08:06.281962 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:08:06.282077 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:08:06.282639 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.283854 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.286605 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.287576 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.287770 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:06.287833 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 16:08:06.288466 (+ 633us) server_negotiation.cc:299] Negotiation successful
0924 16:08:06.288596 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 16:08:06.288951 15957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:06.281311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:45439 (local address 127.3.214.195:35425)
0924 16:08:06.281514 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 16:08:06.281540 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:08:06.281712 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:08:06.282194 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:08:06.282201 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:08:06.282250 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:08:06.282485 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:08:06.282498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.283996 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.284002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:08:06.286418 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:08:06.286432 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.287797 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.287805 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:08:06.288165 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:06.288215 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 16:08:06.288401 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 16:08:06.288593 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 16:08:06.290354 15632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4845bab6fc0a4d5fbdf32666c046dc5b" candidate_uuid: "1927c9f32f1646cca00301db59382ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c14c4cae2e84b44ba5bb11fdabbc4ee" is_pre_election: true
I20260924 16:08:06.291082 15632 raft_consensus.cc:2454] T 4845bab6fc0a4d5fbdf32666c046dc5b P 8c14c4cae2e84b44ba5bb11fdabbc4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1927c9f32f1646cca00301db59382ceb in term 0.
I20260924 16:08:06.291936 15972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:06.281154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:33119 (local address 127.3.214.194:37779)
0924 16:08:06.281385 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 16:08:06.281390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:08:06.283677 (+ 2287us) server_negotiation.cc:407] Connection header received
0924 16:08:06.283847 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:08:06.283851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:08:06.283940 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:08:06.284039 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:08:06.285473 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.286672 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.289474 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.290808 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.290987 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:06.291068 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 16:08:06.291453 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 16:08:06.291595 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 16:08:06.292317 15843 leader_election.cc:302] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1927c9f32f1646cca00301db59382ceb, 8c14c4cae2e84b44ba5bb11fdabbc4ee; no voters:
I20260924 16:08:06.292793 15980 raft_consensus.cc:2781] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:08:06.293056 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:06.281314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:37779 (local address 127.3.214.195:33119)
0924 16:08:06.283539 (+ 2225us) negotiation.cc:107] Waiting for socket to connect
0924 16:08:06.283563 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:08:06.283703 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:08:06.284062 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:08:06.284066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:08:06.284094 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:08:06.285193 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:08:06.285211 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.286900 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.286905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:08:06.289247 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:08:06.289265 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.291068 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.291076 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:08:06.291261 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:06.291294 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 16:08:06.291389 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 16:08:06.291563 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2066,"mutex_wait_us":492,"thread_start_us":639,"threads_started":1}
I20260924 16:08:06.293169 15980 raft_consensus.cc:482] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:08:06.293347 15980 raft_consensus.cc:3037] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:08:06.294299 15768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4845bab6fc0a4d5fbdf32666c046dc5b" candidate_uuid: "1927c9f32f1646cca00301db59382ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72b35e925cd485181a5e2ca83f4c3ad" is_pre_election: true
I20260924 16:08:06.294956 15768 raft_consensus.cc:2454] T 4845bab6fc0a4d5fbdf32666c046dc5b P e72b35e925cd485181a5e2ca83f4c3ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1927c9f32f1646cca00301db59382ceb in term 0.
I20260924 16:08:06.296151 15980 raft_consensus.cc:504] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927c9f32f1646cca00301db59382ceb" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 36921 } } peers { permanent_uuid: "8c14c4cae2e84b44ba5bb11fdabbc4ee" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 45439 } } peers { permanent_uuid: "e72b35e925cd485181a5e2ca83f4c3ad" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 37779 } }
I20260924 16:08:06.296862 15980 leader_election.cc:288] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb [CANDIDATE]: Term 1 election: Requested vote from peers 8c14c4cae2e84b44ba5bb11fdabbc4ee (127.3.214.193:45439), e72b35e925cd485181a5e2ca83f4c3ad (127.3.214.194:37779)
I20260924 16:08:06.297535 15768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4845bab6fc0a4d5fbdf32666c046dc5b" candidate_uuid: "1927c9f32f1646cca00301db59382ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72b35e925cd485181a5e2ca83f4c3ad"
I20260924 16:08:06.297540 15632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4845bab6fc0a4d5fbdf32666c046dc5b" candidate_uuid: "1927c9f32f1646cca00301db59382ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c14c4cae2e84b44ba5bb11fdabbc4ee"
I20260924 16:08:06.297780 15768 raft_consensus.cc:3037] T 4845bab6fc0a4d5fbdf32666c046dc5b P e72b35e925cd485181a5e2ca83f4c3ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:08:06.297793 15632 raft_consensus.cc:3037] T 4845bab6fc0a4d5fbdf32666c046dc5b P 8c14c4cae2e84b44ba5bb11fdabbc4ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:08:06.300401 15768 raft_consensus.cc:2454] T 4845bab6fc0a4d5fbdf32666c046dc5b P e72b35e925cd485181a5e2ca83f4c3ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1927c9f32f1646cca00301db59382ceb in term 1.
I20260924 16:08:06.300573 15632 raft_consensus.cc:2454] T 4845bab6fc0a4d5fbdf32666c046dc5b P 8c14c4cae2e84b44ba5bb11fdabbc4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1927c9f32f1646cca00301db59382ceb in term 1.
I20260924 16:08:06.301019 15844 leader_election.cc:302] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1927c9f32f1646cca00301db59382ceb, e72b35e925cd485181a5e2ca83f4c3ad; no voters:
I20260924 16:08:06.301522 15980 raft_consensus.cc:2781] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:08:06.302227 15980 raft_consensus.cc:686] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb [term 1 LEADER]: Becoming Leader. State: Replica: 1927c9f32f1646cca00301db59382ceb, State: Running, Role: LEADER
I20260924 16:08:06.303293 15980 consensus_queue.cc:237] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927c9f32f1646cca00301db59382ceb" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 36921 } } peers { permanent_uuid: "8c14c4cae2e84b44ba5bb11fdabbc4ee" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 45439 } } peers { permanent_uuid: "e72b35e925cd485181a5e2ca83f4c3ad" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 37779 } }
I20260924 16:08:06.309514 15462 catalog_manager.cc:5495] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb reported cstate change: term changed from 0 to 1, leader changed from <none> to 1927c9f32f1646cca00301db59382ceb (127.3.214.195). New cstate: current_term: 1 leader_uuid: "1927c9f32f1646cca00301db59382ceb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927c9f32f1646cca00301db59382ceb" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 36921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c14c4cae2e84b44ba5bb11fdabbc4ee" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 45439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72b35e925cd485181a5e2ca83f4c3ad" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 37779 } health_report { overall_health: UNKNOWN } } }
W20260924 16:08:06.329216 15955 tablet.cc:2369] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:08:06.343096 15968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:06.337497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59072 (local address 127.3.214.195:36921)
0924 16:08:06.337738 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 16:08:06.337745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:08:06.337769 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:08:06.337917 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:08:06.337921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:08:06.338009 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:08:06.338197 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:08:06.338581 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.339327 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.340845 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.341245 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.341348 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:06.341913 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:08:06.342513 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:08:06.342652 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 16:08:06.342752 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 16:08:06.342837 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 16:08:06.345099 15814 tablet.cc:2369] T 4845bab6fc0a4d5fbdf32666c046dc5b P e72b35e925cd485181a5e2ca83f4c3ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:08:06.353585 15768 raft_consensus.cc:1261] T 4845bab6fc0a4d5fbdf32666c046dc5b P e72b35e925cd485181a5e2ca83f4c3ad [term 1 FOLLOWER]: Refusing update from remote peer 1927c9f32f1646cca00301db59382ceb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:08:06.353650 15632 raft_consensus.cc:1261] T 4845bab6fc0a4d5fbdf32666c046dc5b P 8c14c4cae2e84b44ba5bb11fdabbc4ee [term 1 FOLLOWER]: Refusing update from remote peer 1927c9f32f1646cca00301db59382ceb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:08:06.354913 15983 consensus_queue.cc:1035] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c14c4cae2e84b44ba5bb11fdabbc4ee" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 45439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:08:06.355345 15980 consensus_queue.cc:1035] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72b35e925cd485181a5e2ca83f4c3ad" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 37779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:08:06.386426 15462 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:08:06.386811 15462 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:08:06.391609 15462 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 627ee5b3d4524c6f82e65bfe2f920822: Sending DeleteTablet for 3 replicas of tablet 4845bab6fc0a4d5fbdf32666c046dc5b
I20260924 16:08:06.393461 15748 tablet_service.cc:1514] Processing DeleteTablet for tablet 4845bab6fc0a4d5fbdf32666c046dc5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:08:06 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:37444
I20260924 16:08:06.393472 15889 tablet_service.cc:1514] Processing DeleteTablet for tablet 4845bab6fc0a4d5fbdf32666c046dc5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:08:06 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:59062
I20260924 16:08:06.393500 15612 tablet_service.cc:1514] Processing DeleteTablet for tablet 4845bab6fc0a4d5fbdf32666c046dc5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:08:06 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:33598
I20260924 16:08:06.402400 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 15544
I20260924 16:08:06.407313 15997 tablet_replica.cc:330] T 4845bab6fc0a4d5fbdf32666c046dc5b P e72b35e925cd485181a5e2ca83f4c3ad: stopping tablet replica
I20260924 16:08:06.408198 15996 tablet_replica.cc:330] T 4845bab6fc0a4d5fbdf32666c046dc5b P 8c14c4cae2e84b44ba5bb11fdabbc4ee: stopping tablet replica
I20260924 16:08:06.408890 15996 raft_consensus.cc:2229] T 4845bab6fc0a4d5fbdf32666c046dc5b P 8c14c4cae2e84b44ba5bb11fdabbc4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:08:06.409529 15997 raft_consensus.cc:2229] T 4845bab6fc0a4d5fbdf32666c046dc5b P e72b35e925cd485181a5e2ca83f4c3ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:08:06.410265 15996 raft_consensus.cc:2258] T 4845bab6fc0a4d5fbdf32666c046dc5b P 8c14c4cae2e84b44ba5bb11fdabbc4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:08:06.410821 15997 raft_consensus.cc:2258] T 4845bab6fc0a4d5fbdf32666c046dc5b P e72b35e925cd485181a5e2ca83f4c3ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:08:06.414268 15996 ts_tablet_manager.cc:1905] T 4845bab6fc0a4d5fbdf32666c046dc5b P 8c14c4cae2e84b44ba5bb11fdabbc4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:08:06.414387 15997 ts_tablet_manager.cc:1905] T 4845bab6fc0a4d5fbdf32666c046dc5b P e72b35e925cd485181a5e2ca83f4c3ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:08:06.418571 15998 tablet_replica.cc:330] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb: stopping tablet replica
I20260924 16:08:06.419183 15998 raft_consensus.cc:2229] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb [term 1 LEADER]: Raft consensus shutting down.
Sep 24 16:08:06 dist-test-slave-1gc3 krb5kdc[14984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.193@KRBTEST.COM
I20260924 16:08:06.419965 15998 raft_consensus.cc:2258] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:08:06.422173 15998 ts_tablet_manager.cc:1905] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:08:06.422194 15996 ts_tablet_manager.cc:1918] T 4845bab6fc0a4d5fbdf32666c046dc5b P 8c14c4cae2e84b44ba5bb11fdabbc4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:08:06.422385 15996 log.cc:1196] T 4845bab6fc0a4d5fbdf32666c046dc5b P 8c14c4cae2e84b44ba5bb11fdabbc4ee: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-0/wal/wals/4845bab6fc0a4d5fbdf32666c046dc5b
I20260924 16:08:06.423029 15996 ts_tablet_manager.cc:1939] T 4845bab6fc0a4d5fbdf32666c046dc5b P 8c14c4cae2e84b44ba5bb11fdabbc4ee: Deleting consensus metadata
I20260924 16:08:06.423098 15997 ts_tablet_manager.cc:1918] T 4845bab6fc0a4d5fbdf32666c046dc5b P e72b35e925cd485181a5e2ca83f4c3ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:08:06.423620 15997 log.cc:1196] T 4845bab6fc0a4d5fbdf32666c046dc5b P e72b35e925cd485181a5e2ca83f4c3ad: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/4845bab6fc0a4d5fbdf32666c046dc5b
I20260924 16:08:06.424254 15997 ts_tablet_manager.cc:1939] T 4845bab6fc0a4d5fbdf32666c046dc5b P e72b35e925cd485181a5e2ca83f4c3ad: Deleting consensus metadata
I20260924 16:08:06.425295 15449 catalog_manager.cc:4841] TS 8c14c4cae2e84b44ba5bb11fdabbc4ee (127.3.214.193:45439): tablet 4845bab6fc0a4d5fbdf32666c046dc5b (table test-table [id=d664e5740fbe49cb8147b04d91710237]) successfully deleted
I20260924 16:08:06.428023 15450 catalog_manager.cc:4841] TS e72b35e925cd485181a5e2ca83f4c3ad (127.3.214.194:37779): tablet 4845bab6fc0a4d5fbdf32666c046dc5b (table test-table [id=d664e5740fbe49cb8147b04d91710237]) successfully deleted
I20260924 16:08:06.429572 15998 ts_tablet_manager.cc:1918] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:08:06.429852 15998 log.cc:1196] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/4845bab6fc0a4d5fbdf32666c046dc5b
I20260924 16:08:06.429854 15971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:06.404181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33612 (local address 127.3.214.193:45439)
0924 16:08:06.404440 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 16:08:06.404448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:08:06.407723 (+ 3275us) server_negotiation.cc:407] Connection header received
0924 16:08:06.408117 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:08:06.408127 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:08:06.408250 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:08:06.408333 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:08:06.410639 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.411998 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.414038 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.414736 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.414840 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:06.420970 (+ 6130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:08:06.420995 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:08:06.421020 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:08:06.421085 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:08:06.425580 (+ 4495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:08:06.426871 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:08:06.426881 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:08:06.426885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:08:06.427069 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:08:06.427549 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:08:06.427554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:08:06.427557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:08:06.428131 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:08:06.428440 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 16:08:06.429053 (+ 613us) server_negotiation.cc:299] Negotiation successful
0924 16:08:06.429347 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 16:08:06.430487 15998 ts_tablet_manager.cc:1939] T 4845bab6fc0a4d5fbdf32666c046dc5b P 1927c9f32f1646cca00301db59382ceb: Deleting consensus metadata
I20260924 16:08:06.430632 15672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:08:06.432900 15448 catalog_manager.cc:4841] TS 1927c9f32f1646cca00301db59382ceb (127.3.214.195:36921): tablet 4845bab6fc0a4d5fbdf32666c046dc5b (table test-table [id=d664e5740fbe49cb8147b04d91710237]) successfully deleted
I20260924 16:08:06.586814 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 15544
I20260924 16:08:06.609750 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 15683
Sep 24 16:08:06 dist-test-slave-1gc3 krb5kdc[14984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.194@KRBTEST.COM
I20260924 16:08:06.627650 15972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:06.610801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37454 (local address 127.3.214.194:37779)
0924 16:08:06.611011 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 16:08:06.611018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:08:06.611118 (+ 100us) server_negotiation.cc:407] Connection header received
0924 16:08:06.611337 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:08:06.611343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:08:06.611450 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:08:06.611602 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:08:06.613031 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.613838 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.615403 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.615828 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.615950 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:06.620669 (+ 4719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:08:06.620684 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:08:06.620705 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:08:06.620753 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:08:06.624275 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:08:06.625084 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:08:06.625093 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:08:06.625095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:08:06.625256 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:08:06.625656 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:08:06.625660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:08:06.625662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:08:06.626218 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:08:06.626695 (+ 477us) server_negotiation.cc:1035] Waiting for connection context
0924 16:08:06.626930 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 16:08:06.627204 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 16:08:06.628492 15808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:08:06.731195 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 15683
I20260924 16:08:06.750558 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 15826
Sep 24 16:08:06 dist-test-slave-1gc3 krb5kdc[14984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.195@KRBTEST.COM
I20260924 16:08:06.768834 15968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:06.751608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59086 (local address 127.3.214.195:36921)
0924 16:08:06.751825 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 16:08:06.751831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:08:06.751881 (+ 50us) server_negotiation.cc:407] Connection header received
0924 16:08:06.752152 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:08:06.752157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:08:06.752283 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:08:06.752383 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:08:06.753836 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.754759 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.756179 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:06.756653 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:06.756769 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:06.761784 (+ 5015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:08:06.761804 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:08:06.761832 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:08:06.761892 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:08:06.765607 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:08:06.766485 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:08:06.766493 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:08:06.766496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:08:06.766658 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:08:06.767028 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:08:06.767031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:08:06.767033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:08:06.767482 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:08:06.767826 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0924 16:08:06.768146 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 16:08:06.768400 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 16:08:06.769809 15949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:08:06.914587 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 15826
I20260924 16:08:06.935310 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 15432
I20260924 16:08:06.936836 15494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:08:07.058207 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 15432
I20260924 16:08:07.085620 3931 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 16:08:07.433070 3931 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 16:08:07.433327 3931 mini_ranger.cc:66] Stopping Ranger...
I20260924 16:08:07.781412 3931 mini_ranger.cc:68] Stopped Ranger
2026-09-24 16:08:07.781 UTC [15014] LOG: received smart shutdown request
2026-09-24 16:08:07.785 UTC [15014] LOG: background worker "logical replication launcher" (PID 15023) exited with exit code 1
2026-09-24 16:08:07.786 UTC [15018] LOG: shutting down
2026-09-24 16:08:07.979 UTC [15014] LOG: database system is shut down
2026-09-24T16:08:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (58183 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:08:08 dist-test-slave-1gc3 krb5kdc[16012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:08:08 dist-test-slave-1gc3 krb5kdc[16012](info): set up 2 sockets
Sep 24 16:08:08 dist-test-slave-1gc3 krb5kdc[16012](info): commencing operation
krb5kdc: starting...
W20260924 16:08:09.172574 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:08:09 dist-test-slave-1gc3 krb5kdc[16012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:08:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:08:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.214.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.214.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.214.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/rangeradmin_127.3.214.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.214.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/rangeradmin_127.3.214.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.214.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.214.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.214.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/rangerlookup_127.3.214.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.214.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/rangerlookup_127.3.214.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.214.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.214.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.214.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/HTTP_127.3.214.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.214.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/HTTP_127.3.214.212@KRBTEST.COM.keytab.
I20260924 16:08:09.585497 3931 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHlT5a5/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/postgres -l logfile start
2026-09-24 16:08:12.324 UTC [16042] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 16:08:12.324 UTC [16042] LOG: listening on IPv4 address "127.3.214.212", port 35891
2026-09-24 16:08:12.327 UTC [16042] LOG: listening on Unix socket "/tmp/.s.PGSQL.35891"
2026-09-24 16:08:12.351 UTC [16044] LOG: database system was shut down at 2026-09-24 16:08:10 UTC
2026-09-24 16:08:12.355 UTC [16042] LOG: database system is ready to accept connections
I20260924 16:08:13.267585 3931 mini_postgres.cc:96] Postgres bound to 35891
2026-09-24 16:08:13.289 UTC [16053] FATAL: database "slave" does not exist
127.3.214.212:35891 - accepting connections
I20260924 16:08:13.290766 3931 mini_ranger.cc:161] Starting Ranger...
I20260924 16:08:13.322979 3931 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 16:08:13.858569 3931 mini_ranger.cc:87] Created ranger Postgres database
I20260924 16:08:13.858685 3931 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ranger-admin
2026-09-24 16:08:14,168 [I] DB FLAVOR :POSTGRES
2026-09-24 16:08:14,169 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 16:08:14,169 [I] Checking connection..
2026-09-24 16:08:14,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:35891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 16:08:14,450 [I] Checking connection passed.
2026-09-24 16:08:14,451 [I] --------- Verifying version history table ---------
2026-09-24 16:08:14,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:35891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:08:14,729 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 16:08:14,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:35891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:08:15,035 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 16:08:15,036 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 16:08:15,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:35891/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 16:08:15.327 UTC [16117] WARNING: there is no transaction in progress
2026-09-24 16:08:15,336 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 16:08:15,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:35891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:08:15,627 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 16:08:15,627 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 16:08:15,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:35891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 16:08:15,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:35891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 16:08:16,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:35891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1gc3.c.gcp-upstream.internal','N') ;"
2026-09-24 16:08:16,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:35891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 16:08:16,727 [I] Table x_portal_user does not exist in database ranger
2026-09-24 16:08:16,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:35891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 16:08:16,998 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 16:08:16,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:35891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 16:08:17,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:35891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 16:08:17,553 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 16:08:17,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:35891/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 16:08:18.695 UTC [16225] WARNING: there is no transaction in progress
2026-09-24 16:08:18.714 UTC [16225] WARNING: there is no transaction in progress
2026-09-24 16:08:18.733 UTC [16225] WARNING: there is no transaction in progress
2026-09-24 16:08:18.752 UTC [16225] WARNING: there is no transaction in progress
2026-09-24 16:08:18.773 UTC [16225] WARNING: there is no transaction in progress
2026-09-24 16:08:18.797 UTC [16225] WARNING: there is no transaction in progress
2026-09-24 16:08:18.870 UTC [16225] WARNING: there is no transaction in progress
2026-09-24 16:08:18.880 UTC [16225] WARNING: there is no transaction in progress
2026-09-24 16:08:18.897 UTC [16225] WARNING: there is no transaction in progress
2026-09-24 16:08:18.909 UTC [16225] WARNING: there is no transaction in progress
2026-09-24 16:08:18.922 UTC [16225] WARNING: there is no transaction in progress
2026-09-24 16:08:18.940 UTC [16225] WARNING: there is no transaction in progress
2026-09-24 16:08:18.952 UTC [16225] WARNING: there is no transaction in progress
2026-09-24 16:08:18.961 UTC [16225] WARNING: there is no transaction in progress
2026-09-24 16:08:18.970 UTC [16225] WARNING: there is no transaction in progress
2026-09-24 16:08:18.981 UTC [16225] WARNING: there is no transaction in progress
2026-09-24 16:08:19,791 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 16:08:19,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:35891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 16:08:20,064 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 16:08:20,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:35891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 16:08:20,330 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 16:08:20,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:35891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 16:08:20,607 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 16:08:20,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:35891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 16:08:20,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:35891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 16:08:21,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:35891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1gc3.c.gcp-upstream.internal';"
2026-09-24 16:08:21,446 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 16:08:21,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:35891/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 16:08:21,748 [I] DB_PATCHES have already been applied
I20260924 16:08:21.755769 3931 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHlT5a5/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 16:08:21.755880 3931 mini_ranger.cc:193] Using host: 127.3.214.212
I20260924 16:08:21.777191 3931 mini_ranger.cc:227] Ranger admin URL: http://127.3.214.212:54225
Sep 24, 2026 4:08:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:08:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:08:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 4:08:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 4:08:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 4:08:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 4:08:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 16:08:22.852133 3931 mini_ranger.cc:160] Time spent starting Ranger: real 9.561s user 0.002s sys 0.053s
Sep 24, 2026 4:08:44 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 4:08:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 4:08:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 4:08:44 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 4:08:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 4:08:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 4:08:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 4:08:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 4:08:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 4:08:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 4:08:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 4:08:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 4:08:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 4:08:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 4:08:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 4:08:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 4:08:44 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 4:08:45 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 16:08:46.601902 3931 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.214.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.214.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.214.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/rangerkms_127.3.214.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.214.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/rangerkms_127.3.214.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.214.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.214.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.214.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/HTTP_127.3.214.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.214.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/HTTP_127.3.214.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 16:08:46 dist-test-slave-1gc3 krb5kdc[16012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266126, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 16:08:46.817863 3931 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 16:08:46.843916 3931 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 16:08:47.806021 3931 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 16:08:47.806190 3931 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ranger-kms
I20260924 16:08:47.806234 3931 mini_ranger_kms.cc:227] Using postgres at 127.3.214.212:35891
2026-09-24 16:08:48,192 [I] DB FLAVOR :POSTGRES
2026-09-24 16:08:48,192 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 16:08:48,192 [I] Checking connection
2026-09-24 16:08:48,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:35891/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 16:08:48,473 [I] connection success
2026-09-24 16:08:48,473 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 16:08:48,473 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 16:08:48,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:35891/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 16:08:48,751 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 16:08:48,751 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 16:08:48,751 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 16:08:48,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.214.212:35891/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 16:08:49,035 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 16:08:49.522918 3931 mini_ranger_kms.cc:310] Created kms service
I20260924 16:08:49.756012 3931 mini_ranger_kms.cc:326] Created kudu user
I20260924 16:08:49.802493 3931 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 16:08:49 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266126, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
I20260924 16:08:50.269186 3931 mini_ranger_kms.cc:384] Added ranger policy
I20260924 16:08:50.269618 3931 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ranger-kms:/tmp/dist-test-taskHlT5a5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskHlT5a5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 16:08:50.269685 3931 mini_ranger_kms.cc:240] Using host: 127.3.214.212
I20260924 16:08:50.294622 3931 mini_ranger_kms.cc:276] Ranger KMS PID: 16407
I20260924 16:08:50.294801 3931 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.214.212:33379
Sep 24, 2026 4:08:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:08:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:08:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 4:08:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 4:08:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 4:08:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 16:08:52.445870 3931 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.628s user 0.079s sys 0.337s
I20260924 16:08:52.446179 3931 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 16:08:52.446376 3931 mini_ranger_kms.cc:401] 127.3.214.212:33379/kms/v1/keys
Sep 24 16:08:52 dist-test-slave-1gc3 krb5kdc[16012](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790266132, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.214.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 16:08:52 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266132, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.214.212@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
Sep 24 16:08:53 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266132, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.214.212@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
Sep 24 16:08:55 dist-test-slave-1gc3 krb5kdc[16012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:08:56.000617 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:38275
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:40065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:38275
--ranger_config_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.214.212:33379/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:08:56.408876 16457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:08:56.409302 16457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:08:56.409417 16457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:08:56.417856 16457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:08:56.418008 16457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:08:56.418057 16457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:08:56.418147 16457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:08:56.418228 16457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:08:56.430007 16457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:40065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.214.212:33379/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:38275
--ranger_config_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:38275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:08:56.431838 16457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:08:56.433737 16457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:08:56.449085 16462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:08:56.449187 16465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:08:56.449385 16457 server_base.cc:1034] running on GCE node
W20260924 16:08:56.449039 16463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:08:56.450949 16457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:08:56.452780 16457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:08:56.454073 16457 hybrid_clock.cc:648] HybridClock initialized: now 1790266136454015 us; error 47 us; skew 500 ppm
I20260924 16:08:56.454630 16457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:08:56 dist-test-slave-1gc3 krb5kdc[16012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266136, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:08:56.464056 16457 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:08:56.466187 16457 webserver.cc:460] Webserver started at http://127.3.214.254:37477/ using document root <none> and password file <none>
I20260924 16:08:56.467011 16457 fs_manager.cc:359] Metadata directory not provided
I20260924 16:08:56.467136 16457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:08:56.467471 16457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:08:56 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266136, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
I20260924 16:08:56.528404 16457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "3be9e337e5f94c96a4c6f6f372f6298a"
format_stamp: "Formatted at 2026-09-24 16:08:56 on dist-test-slave-1gc3"
server_key: "8efb7f7fd22e4e56f54001dab8297192"
server_key_iv: "925d78844478a5a2f35aeb5aca8ca7e6"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:56.529721 16457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "3be9e337e5f94c96a4c6f6f372f6298a"
format_stamp: "Formatted at 2026-09-24 16:08:56 on dist-test-slave-1gc3"
server_key: "8efb7f7fd22e4e56f54001dab8297192"
server_key_iv: "925d78844478a5a2f35aeb5aca8ca7e6"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:56.538383 16457 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260924 16:08:56.563555 16472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:08:56.565609 16457 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:08:56.565862 16457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "3be9e337e5f94c96a4c6f6f372f6298a"
format_stamp: "Formatted at 2026-09-24 16:08:56 on dist-test-slave-1gc3"
server_key: "8efb7f7fd22e4e56f54001dab8297192"
server_key_iv: "925d78844478a5a2f35aeb5aca8ca7e6"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:56.566176 16457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:08:56.593753 16457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:08:56.599634 16457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:08:56.600003 16457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:08:56.626559 16457 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:38275
I20260924 16:08:56.626583 16524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:38275 every 8 connection(s)
I20260924 16:08:56.628468 16457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:08:56.628934 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 16457
I20260924 16:08:56.629185 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:08:56.629957 3931 external_mini_cluster.cc:1428] Setting key a4d15555f804647cdf6a2bf092035bb8
I20260924 16:08:56.636420 16525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:08:56.660565 16525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a: Bootstrap starting.
I20260924 16:08:56.665767 16525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a: Neither blocks nor log segments found. Creating new log.
I20260924 16:08:56.667297 16525 log.cc:824] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:08:56 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:08:56.672009 16525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a: No bootstrap required, opened a new log
I20260924 16:08:56.681100 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:56.638592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48800 (local address 127.3.214.254:38275)
0924 16:08:56.640461 (+ 1869us) server_negotiation.cc:206] Beginning negotiation
0924 16:08:56.640483 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 16:08:56.651115 (+ 10632us) server_negotiation.cc:407] Connection header received
0924 16:08:56.655035 (+ 3920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:08:56.655091 (+ 56us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:08:56.655921 (+ 830us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:08:56.656586 (+ 665us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:08:56.658564 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:56.660805 (+ 2241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:56.663265 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:56.664658 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:56.664935 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:56.670528 (+ 5593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:08:56.670541 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:08:56.670565 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:08:56.670620 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:08:56.674897 (+ 4277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:08:56.675941 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:08:56.675951 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:08:56.675966 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:08:56.676167 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:08:56.677129 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:08:56.677134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:08:56.677136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:08:56.677963 (+ 827us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:08:56.678441 (+ 478us) server_negotiation.cc:1035] Waiting for connection context
0924 16:08:56.679016 (+ 575us) server_negotiation.cc:299] Negotiation successful
0924 16:08:56.679628 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1468,"thread_start_us":1118,"threads_started":1}
I20260924 16:08:56.684425 16525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be9e337e5f94c96a4c6f6f372f6298a" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38275 } }
I20260924 16:08:56.685245 16525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:08:56.685379 16525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3be9e337e5f94c96a4c6f6f372f6298a, State: Initialized, Role: FOLLOWER
I20260924 16:08:56.686236 16525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be9e337e5f94c96a4c6f6f372f6298a" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38275 } }
I20260924 16:08:56.686568 16525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:08:56.686731 16525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:08:56.686931 16525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:08:56.690285 16525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be9e337e5f94c96a4c6f6f372f6298a" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38275 } }
I20260924 16:08:56.691108 16525 leader_election.cc:302] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3be9e337e5f94c96a4c6f6f372f6298a; no voters:
I20260924 16:08:56.691928 16525 leader_election.cc:288] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:08:56.692160 16530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:08:56.693305 16530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [term 1 LEADER]: Becoming Leader. State: Replica: 3be9e337e5f94c96a4c6f6f372f6298a, State: Running, Role: LEADER
I20260924 16:08:56.694286 16530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be9e337e5f94c96a4c6f6f372f6298a" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38275 } }
I20260924 16:08:56.694797 16525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:08:56.698453 16532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3be9e337e5f94c96a4c6f6f372f6298a. Latest consensus state: current_term: 1 leader_uuid: "3be9e337e5f94c96a4c6f6f372f6298a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be9e337e5f94c96a4c6f6f372f6298a" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38275 } } }
I20260924 16:08:56.698982 16532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [sys.catalog]: This master's current role is: LEADER
I20260924 16:08:56.699698 16531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3be9e337e5f94c96a4c6f6f372f6298a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be9e337e5f94c96a4c6f6f372f6298a" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38275 } } }
I20260924 16:08:56.700084 16531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [sys.catalog]: This master's current role is: LEADER
I20260924 16:08:56.712237 16537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:08:56.725180 16537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:08:56.747956 16525 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 16:08:56.749747 16537 catalog_manager.cc:1347] Generated new cluster ID: 182fad9526c94a8a9d91815a4a2f92d6
I20260924 16:08:56.750078 16537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:08:56.803748 16537 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:08:56.805271 16537 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:08:56.816583 16537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a: Generated new TSK 0
I20260924 16:08:56.818181 16537 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.214.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
I20260924 16:08:57.679172 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.214.212:33379/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:38275
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:40065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:08:57 dist-test-slave-1gc3 krb5kdc[16012](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790266137, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 16:08:58.194832 16570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:08:58.195533 16570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:08:58.195788 16570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:08:58.208637 16570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:08:58.208990 16570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:08:58.209261 16570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:08:58.226639 16570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:40065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.214.212:33379/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:38275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:08:58.228981 16570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:08:58.231176 16570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:08:58.251992 16580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:08:58.256399 16581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:08:58.263302 16583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:08:58.264242 16570 server_base.cc:1034] running on GCE node
I20260924 16:08:58.265339 16570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:08:58.267798 16570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:08:58.269189 16570 hybrid_clock.cc:648] HybridClock initialized: now 1790266138269033 us; error 134 us; skew 500 ppm
I20260924 16:08:58.269788 16570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:08:58 dist-test-slave-1gc3 krb5kdc[16012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266138, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:08:58.274565 16570 init.cc:377] Logged in from keytab as kudu/127.3.214.193@KRBTEST.COM (short username kudu)
I20260924 16:08:58.277055 16570 webserver.cc:460] Webserver started at http://127.3.214.193:38295/ using document root <none> and password file <none>
I20260924 16:08:58.278010 16570 fs_manager.cc:359] Metadata directory not provided
I20260924 16:08:58.278265 16570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:08:58.278789 16570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:08:58 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266138, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
I20260924 16:08:58.308192 16570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "232cdf436bfb45588d65edbd00c4e236"
format_stamp: "Formatted at 2026-09-24 16:08:58 on dist-test-slave-1gc3"
server_key: "46ab8abcf4b42f89c5e69d7e14555322"
server_key_iv: "0edf38c6bfdbfe5accee55090d204317"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:58.309610 16570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "232cdf436bfb45588d65edbd00c4e236"
format_stamp: "Formatted at 2026-09-24 16:08:58 on dist-test-slave-1gc3"
server_key: "46ab8abcf4b42f89c5e69d7e14555322"
server_key_iv: "0edf38c6bfdbfe5accee55090d204317"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:58.318586 16570 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260924 16:08:58.363013 16590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:08:58.367861 16570 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.004s sys 0.004s
I20260924 16:08:58.368407 16570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "232cdf436bfb45588d65edbd00c4e236"
format_stamp: "Formatted at 2026-09-24 16:08:58 on dist-test-slave-1gc3"
server_key: "46ab8abcf4b42f89c5e69d7e14555322"
server_key_iv: "0edf38c6bfdbfe5accee55090d204317"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:58.368916 16570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 24 16:08:58 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266137, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
I20260924 16:08:58.425685 16570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:08:58.433022 16570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:08:58.433600 16570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:08:58.441476 16570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:08:58.443819 16570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:08:58.443944 16570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:08:58.444263 16570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:08:58.444453 16570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:08:58.496237 16570 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:43827
I20260924 16:08:58.496256 16703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:43827 every 8 connection(s)
I20260924 16:08:58.498358 16570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:08:58.499675 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 16570
I20260924 16:08:58.499951 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:08:58.500910 3931 external_mini_cluster.cc:1428] Setting key 6c81a096de9e05a3efccb7543e7f7908
Sep 24 16:08:58 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266138, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.214.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.194@KRBTEST.COM" created.
I20260924 16:08:58.566985 16707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:58.517837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38275 (local address 127.3.214.193:58639)
0924 16:08:58.531382 (+ 13545us) negotiation.cc:107] Waiting for socket to connect
0924 16:08:58.531510 (+ 128us) client_negotiation.cc:174] Beginning negotiation
0924 16:08:58.534846 (+ 3336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:08:58.539019 (+ 4173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:08:58.539036 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:08:58.539828 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:08:58.542765 (+ 2937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:08:58.542824 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:08:58.544706 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:08:58.544712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:08:58.547388 (+ 2676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:08:58.547441 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:08:58.548907 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:08:58.548915 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:08:58.549270 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:58.550191 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:08:58.550227 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:08:58.553975 (+ 3748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:08:58.560103 (+ 6128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:08:58.560118 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:08:58.560149 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:08:58.561214 (+ 1065us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:08:58.561975 (+ 761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:08:58.561982 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:08:58.561986 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:08:58.562460 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:08:58.563646 (+ 1186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:08:58.563660 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:08:58.564256 (+ 596us) client_negotiation.cc:769] Sending connection context
0924 16:08:58.564633 (+ 377us) client_negotiation.cc:240] Negotiation successful
0924 16:08:58.565273 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12953,"thread_start_us":12481,"threads_started":1}
I20260924 16:08:58.568475 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:58.517157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:58639 (local address 127.3.214.254:38275)
0924 16:08:58.537986 (+ 20829us) server_negotiation.cc:206] Beginning negotiation
0924 16:08:58.537991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:08:58.538016 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:08:58.538092 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:08:58.538132 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:08:58.538268 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:08:58.538401 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:08:58.543159 (+ 4758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:58.544396 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:58.547705 (+ 3309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:58.548375 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:58.548513 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:58.554390 (+ 5877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:08:58.554399 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:08:58.554404 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:08:58.554475 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:08:58.559748 (+ 5273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:08:58.561551 (+ 1803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:08:58.561559 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:08:58.561563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:08:58.561791 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:08:58.562758 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:08:58.562768 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:08:58.562772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:08:58.563267 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:08:58.563484 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 16:08:58.564934 (+ 1450us) server_negotiation.cc:299] Negotiation successful
0924 16:08:58.568107 (+ 3173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20667,"thread_start_us":20504,"threads_started":1}
I20260924 16:08:58.570700 16704 heartbeater.cc:347] Connected to a master server at 127.3.214.254:38275
I20260924 16:08:58.571082 16704 heartbeater.cc:464] Registering TS with master...
I20260924 16:08:58.572346 16704 heartbeater.cc:511] Master 127.3.214.254:38275 requested a full tablet report, sending...
I20260924 16:08:58.575951 16490 ts_manager.cc:194] Registered new tserver with Master: 232cdf436bfb45588d65edbd00c4e236 (127.3.214.193:43827)
I20260924 16:08:58.578500 16490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:58639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
W20260924 16:08:58.618283 3931 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.214.194: real 0.113s user 0.005s sys 0.025s
I20260924 16:08:58.619565 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.214.212:33379/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:38275
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:40065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:08:58 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266137, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
W20260924 16:08:59.193130 16710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:08:59.193508 16710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:08:59.193653 16710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:08:59.201875 16710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:08:59.202023 16710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:08:59.202214 16710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:08:59.213502 16710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:40065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.214.212:33379/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:38275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:08:59.215337 16710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:08:59.217042 16710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:08:59.234330 16725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:08:59.234524 16727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:08:59.234736 16724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:08:59.240083 16710 server_base.cc:1034] running on GCE node
I20260924 16:08:59.241026 16710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:08:59.242399 16710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:08:59.243656 16710 hybrid_clock.cc:648] HybridClock initialized: now 1790266139243584 us; error 61 us; skew 500 ppm
I20260924 16:08:59.244064 16710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:08:59 dist-test-slave-1gc3 krb5kdc[16012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266139, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:08:59.247874 16710 init.cc:377] Logged in from keytab as kudu/127.3.214.194@KRBTEST.COM (short username kudu)
I20260924 16:08:59.250022 16710 webserver.cc:460] Webserver started at http://127.3.214.194:38305/ using document root <none> and password file <none>
I20260924 16:08:59.250967 16710 fs_manager.cc:359] Metadata directory not provided
I20260924 16:08:59.251089 16710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:08:59.251463 16710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:08:59 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266139, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
I20260924 16:08:59.272804 16710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "d51791d775cf4d2db0a1213426a70260"
format_stamp: "Formatted at 2026-09-24 16:08:59 on dist-test-slave-1gc3"
server_key: "b0c4229c739a80f9750b7eb56abeb08b"
server_key_iv: "248a579fe8c79aed9698a5b622220164"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:59.273972 16710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "d51791d775cf4d2db0a1213426a70260"
format_stamp: "Formatted at 2026-09-24 16:08:59 on dist-test-slave-1gc3"
server_key: "b0c4229c739a80f9750b7eb56abeb08b"
server_key_iv: "248a579fe8c79aed9698a5b622220164"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:59.281622 16710 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260924 16:08:59.306942 16734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:08:59.309397 16710 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.003s
I20260924 16:08:59.309765 16710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "d51791d775cf4d2db0a1213426a70260"
format_stamp: "Formatted at 2026-09-24 16:08:59 on dist-test-slave-1gc3"
server_key: "b0c4229c739a80f9750b7eb56abeb08b"
server_key_iv: "248a579fe8c79aed9698a5b622220164"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:59.310200 16710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:08:59.329901 16710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:08:59.336412 16710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:08:59.336920 16710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:08:59.339052 16710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:08:59.341441 16710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:08:59.341635 16710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 16:08:59.341818 16710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:08:59.341909 16710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:08:59.398762 16710 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:41255
I20260924 16:08:59.398833 16847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:41255 every 8 connection(s)
I20260924 16:08:59.400839 16710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:08:59.401314 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 16710
I20260924 16:08:59.401515 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:08:59.402384 3931 external_mini_cluster.cc:1428] Setting key 9aee08b659b0aad35f21549f40949aa1
Sep 24 16:08:59 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266139, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.214.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.195@KRBTEST.COM" created.
I20260924 16:08:59.449946 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:59.406685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:51215 (local address 127.3.214.254:38275)
0924 16:08:59.414595 (+ 7910us) server_negotiation.cc:206] Beginning negotiation
0924 16:08:59.414603 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:08:59.418737 (+ 4134us) server_negotiation.cc:407] Connection header received
0924 16:08:59.418842 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:08:59.418847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:08:59.418968 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:08:59.419075 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:08:59.426419 (+ 7344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:59.427654 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:59.430610 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:08:59.431293 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:08:59.431417 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:59.437760 (+ 6343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:08:59.437770 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:08:59.437776 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:08:59.437835 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:08:59.443958 (+ 6123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:08:59.446296 (+ 2338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:08:59.446304 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:08:59.446307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:08:59.446506 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:08:59.447591 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:08:59.447597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:08:59.447599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:08:59.448020 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:08:59.448164 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 16:08:59.449295 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0924 16:08:59.449603 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7738,"thread_start_us":9619,"threads_started":1}
I20260924 16:08:59.452415 16851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:08:59.407645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38275 (local address 127.3.214.194:51215)
0924 16:08:59.415349 (+ 7704us) negotiation.cc:107] Waiting for socket to connect
0924 16:08:59.415424 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0924 16:08:59.418185 (+ 2761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:08:59.423875 (+ 5690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:08:59.423895 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:08:59.424768 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:08:59.426000 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:08:59.426038 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:08:59.428022 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:08:59.428028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:08:59.430365 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:08:59.430391 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:08:59.431886 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:08:59.431895 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:08:59.432291 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:08:59.433506 (+ 1215us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:08:59.433548 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:08:59.437078 (+ 3530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:08:59.444311 (+ 7233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:08:59.444326 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:08:59.444352 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:08:59.446024 (+ 1672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:08:59.446693 (+ 669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:08:59.446697 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:08:59.446700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:08:59.447015 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:08:59.448268 (+ 1253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:08:59.448276 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:08:59.448790 (+ 514us) client_negotiation.cc:769] Sending connection context
0924 16:08:59.450323 (+ 1533us) client_negotiation.cc:240] Negotiation successful
0924 16:08:59.450921 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7271,"thread_start_us":6977,"threads_started":1}
I20260924 16:08:59.454339 16848 heartbeater.cc:347] Connected to a master server at 127.3.214.254:38275
I20260924 16:08:59.454663 16848 heartbeater.cc:464] Registering TS with master...
I20260924 16:08:59.455523 16848 heartbeater.cc:511] Master 127.3.214.254:38275 requested a full tablet report, sending...
I20260924 16:08:59.457211 16490 ts_manager.cc:194] Registered new tserver with Master: d51791d775cf4d2db0a1213426a70260 (127.3.214.194:41255)
I20260924 16:08:59.458057 16490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:51215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
I20260924 16:08:59.482076 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.214.212:33379/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:38275
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:40065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:08:59.584231 16704 heartbeater.cc:503] Master 127.3.214.254:38275 was elected leader, sending a full tablet report...
W20260924 16:08:59.863637 16854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:08:59.863953 16854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:08:59.864049 16854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:08:59.872241 16854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:08:59.872370 16854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:08:59.872488 16854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:08:59.883975 16854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:40065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.214.212:33379/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:38275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:08:59.885360 16854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:08:59.886787 16854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:08:59.902233 16859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:08:59.902130 16862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:08:59.902180 16860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:08:59.903203 16854 server_base.cc:1034] running on GCE node
I20260924 16:08:59.904053 16854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:08:59.905405 16854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:08:59.906663 16854 hybrid_clock.cc:648] HybridClock initialized: now 1790266139906574 us; error 82 us; skew 500 ppm
I20260924 16:08:59.907073 16854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:08:59 dist-test-slave-1gc3 krb5kdc[16012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266139, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:08:59.910989 16854 init.cc:377] Logged in from keytab as kudu/127.3.214.195@KRBTEST.COM (short username kudu)
I20260924 16:08:59.912974 16854 webserver.cc:460] Webserver started at http://127.3.214.195:39861/ using document root <none> and password file <none>
I20260924 16:08:59.913838 16854 fs_manager.cc:359] Metadata directory not provided
I20260924 16:08:59.913970 16854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:08:59.914378 16854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:08:59 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266139, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
I20260924 16:08:59.933073 16854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "0a2fc5e20c504b918d988a4a5b9d58a8"
format_stamp: "Formatted at 2026-09-24 16:08:59 on dist-test-slave-1gc3"
server_key: "e1f3ff876aecc95db7fdf5ee048d2334"
server_key_iv: "0649179d37bb3bb21375c0ff30142a41"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:59.934016 16854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "0a2fc5e20c504b918d988a4a5b9d58a8"
format_stamp: "Formatted at 2026-09-24 16:08:59 on dist-test-slave-1gc3"
server_key: "e1f3ff876aecc95db7fdf5ee048d2334"
server_key_iv: "0649179d37bb3bb21375c0ff30142a41"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:59.940884 16854 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 16:08:59.962481 16869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:08:59.964745 16854 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260924 16:08:59.965015 16854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "0a2fc5e20c504b918d988a4a5b9d58a8"
format_stamp: "Formatted at 2026-09-24 16:08:59 on dist-test-slave-1gc3"
server_key: "e1f3ff876aecc95db7fdf5ee048d2334"
server_key_iv: "0649179d37bb3bb21375c0ff30142a41"
server_key_version: "kuduclusterkey@0"
I20260924 16:08:59.965332 16854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:08:59.979471 16854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:08:59.985347 16854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:08:59.985700 16854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:08:59.986982 16854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:08:59.988677 16854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:08:59.988767 16854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:08:59.988878 16854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:08:59.988943 16854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:00.028559 16854 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:39365
I20260924 16:09:00.028646 16983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:39365 every 8 connection(s)
I20260924 16:09:00.030200 16854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:09:00.034268 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 16854
I20260924 16:09:00.034580 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:09:00.035362 3931 external_mini_cluster.cc:1428] Setting key cbd9d5ad40c6e3779dd7dfc42ea7091e
Sep 24 16:09:00 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266139, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:09:00.057578 16986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:00.033667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:43995 (local address 127.3.214.254:38275)
0924 16:09:00.034502 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:00.034512 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:00.035785 (+ 1273us) server_negotiation.cc:407] Connection header received
0924 16:09:00.038759 (+ 2974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:00.038766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:00.038928 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:00.039056 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:00.041361 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:00.042282 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:00.044256 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:00.044774 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:00.044903 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:00.049533 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:00.049541 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:00.049544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:00.049600 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:00.053211 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:00.054440 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:00.054447 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:00.054451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:00.054645 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:00.055144 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:00.055147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:00.055149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:00.055489 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:00.055635 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:00.056968 (+ 1333us) server_negotiation.cc:299] Negotiation successful
0924 16:09:00.057243 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":413,"threads_started":1}
I20260924 16:09:00.058990 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:00.034318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38275 (local address 127.3.214.195:43995)
0924 16:09:00.035487 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:00.035574 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:00.038198 (+ 2624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:00.039448 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:00.039461 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:00.040175 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:00.041099 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:00.041127 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:00.042461 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:00.042466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:00.044077 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:00.044095 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:00.045060 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:00.045068 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:00.045367 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:00.046441 (+ 1074us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:00.046477 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:00.049141 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:00.053469 (+ 4328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:00.053481 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:00.053505 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:00.054276 (+ 771us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:00.054782 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:00.054785 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:00.054788 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:00.055028 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:00.055688 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:00.055696 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:00.056133 (+ 437us) client_negotiation.cc:769] Sending connection context
0924 16:09:00.056775 (+ 642us) client_negotiation.cc:240] Negotiation successful
0924 16:09:00.057351 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":463,"threads_started":1}
I20260924 16:09:00.061631 16984 heartbeater.cc:347] Connected to a master server at 127.3.214.254:38275
I20260924 16:09:00.062065 16984 heartbeater.cc:464] Registering TS with master...
I20260924 16:09:00.063035 16984 heartbeater.cc:511] Master 127.3.214.254:38275 requested a full tablet report, sending...
I20260924 16:09:00.065680 16490 ts_manager.cc:194] Registered new tserver with Master: 0a2fc5e20c504b918d988a4a5b9d58a8 (127.3.214.195:39365)
I20260924 16:09:00.067066 16490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:43995
I20260924 16:09:00.068327 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:09:00.070740 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 16570
Sep 24 16:09:00 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.193@KRBTEST.COM
I20260924 16:09:00.091821 16989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:00.072766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57072 (local address 127.3.214.193:43827)
0924 16:09:00.073763 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:00.073773 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:00.073808 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:09:00.073894 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:00.073904 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:00.074249 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:00.074488 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:00.076272 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:00.077543 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:00.079244 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:00.079939 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:00.080079 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:00.084849 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:00.084866 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:00.084890 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:00.084938 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:00.088352 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:00.089276 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:00.089286 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:00.089288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:00.089479 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:00.089891 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:00.089895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:00.089897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:00.090382 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:00.090740 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:00.091055 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 16:09:00.091395 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":432,"threads_started":1}
I20260924 16:09:00.093494 16699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:00.188069 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 16570
I20260924 16:09:00.204711 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 16710
Sep 24 16:09:00 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.194@KRBTEST.COM
I20260924 16:09:00.231161 16991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:00.211364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57712 (local address 127.3.214.194:41255)
0924 16:09:00.211970 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:00.211978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:00.212012 (+ 34us) server_negotiation.cc:407] Connection header received
0924 16:09:00.212260 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:00.212277 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:00.212642 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:00.212840 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:00.214347 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:00.215635 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:00.217372 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:00.218168 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:00.218316 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:00.223838 (+ 5522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:00.223867 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:00.223891 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:00.223955 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:00.227457 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:00.228433 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:00.228443 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:00.228446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:00.228626 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:00.229043 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:00.229047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:00.229049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:00.229530 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:00.229880 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:00.230261 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 16:09:00.230609 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":312,"threads_started":1}
I20260924 16:09:00.233024 16843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:00.321086 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 16710
I20260924 16:09:00.339344 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 16854
Sep 24 16:09:00 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.195@KRBTEST.COM
I20260924 16:09:00.358978 16993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:00.340593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37774 (local address 127.3.214.195:39365)
0924 16:09:00.341426 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:00.341435 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:00.341470 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:09:00.341544 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:00.341551 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:00.341795 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:00.341949 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:00.343559 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:00.344631 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:00.346196 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:00.346866 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:00.346983 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:00.352207 (+ 5224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:00.352226 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:00.352246 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:00.352293 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:00.355524 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:00.356495 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:00.356505 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:00.356508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:00.356678 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:00.357108 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:00.357113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:00.357115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:00.357545 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:00.357901 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:00.358286 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 16:09:00.358598 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":411,"threads_started":1}
I20260924 16:09:00.361050 16978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:00.450587 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 16854
I20260924 16:09:00.468997 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 16457
I20260924 16:09:00.470780 16520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:00.586508 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 16457
I20260924 16:09:00.615940 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:38275
--webserver_interface=127.3.214.254
--webserver_port=37477
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:40065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:38275
--ranger_config_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.214.212:33379/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.214.212:33379/kms with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:09:00.982203 16996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:00.982645 16996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:00.982774 16996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:00.991166 16996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:09:00.991336 16996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:00.991402 16996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:09:00.991492 16996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:09:00.991549 16996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:09:01.004298 16996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:40065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.214.212:33379/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:38275
--ranger_config_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:38275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=37477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:01.006428 16996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:01.008426 16996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:01.024593 17001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:01.024653 17002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:01.024590 17004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:01.025844 16996 server_base.cc:1034] running on GCE node
I20260924 16:09:01.027086 16996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:01.029111 16996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:09:01.030337 16996 hybrid_clock.cc:648] HybridClock initialized: now 1790266141030290 us; error 47 us; skew 500 ppm
I20260924 16:09:01.030821 16996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:01 dist-test-slave-1gc3 krb5kdc[16012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266141, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:01.034791 16996 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:09:01.036978 16996 webserver.cc:460] Webserver started at http://127.3.214.254:37477/ using document root <none> and password file <none>
I20260924 16:09:01.037868 16996 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:01.038031 16996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 16:09:01 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266141, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
I20260924 16:09:01.101970 16996 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:09:01.106003 17012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:01.108109 16996 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260924 16:09:01.108363 16996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "3be9e337e5f94c96a4c6f6f372f6298a"
format_stamp: "Formatted at 2026-09-24 16:08:56 on dist-test-slave-1gc3"
server_key: "8efb7f7fd22e4e56f54001dab8297192"
server_key_iv: "925d78844478a5a2f35aeb5aca8ca7e6"
server_key_version: "kuduclusterkey@0"
I20260924 16:09:01.109570 16996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:01.130441 16996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:01.135412 16996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:01.135742 16996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:01.157462 16996 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:38275
I20260924 16:09:01.157485 17063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:38275 every 8 connection(s)
I20260924 16:09:01.159523 16996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:09:01.167757 17064 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:09:01.168689 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 16996
I20260924 16:09:01.169708 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.214.212:33379/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:43827
--local_ip_for_outbound_sockets=127.3.214.193
--tserver_master_addrs=127.3.214.254:38275
--webserver_port=38295
--webserver_interface=127.3.214.193
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:40065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.214.212:33379/kms with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:09:01.171720 17064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a: Bootstrap starting.
I20260924 16:09:01.191766 17064 log.cc:824] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a: Log is configured to *not* fsync() on all Append() calls
I20260924 16:09:01.201414 17064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:09:01.201967 17064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a: Bootstrap complete.
I20260924 16:09:01.211454 17064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be9e337e5f94c96a4c6f6f372f6298a" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38275 } }
I20260924 16:09:01.212421 17064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3be9e337e5f94c96a4c6f6f372f6298a, State: Initialized, Role: FOLLOWER
I20260924 16:09:01.213126 17064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be9e337e5f94c96a4c6f6f372f6298a" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38275 } }
I20260924 16:09:01.213368 17064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:09:01.213483 17064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:09:01.213667 17064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:09:01.218272 17064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be9e337e5f94c96a4c6f6f372f6298a" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38275 } }
I20260924 16:09:01.218856 17064 leader_election.cc:302] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3be9e337e5f94c96a4c6f6f372f6298a; no voters:
I20260924 16:09:01.219482 17064 leader_election.cc:288] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 16:09:01.219650 17068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [term 2 FOLLOWER]: Leader election won for term 2
I20260924 16:09:01.222034 17064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:09:01.226591 17068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [term 2 LEADER]: Becoming Leader. State: Replica: 3be9e337e5f94c96a4c6f6f372f6298a, State: Running, Role: LEADER
I20260924 16:09:01.227460 17068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be9e337e5f94c96a4c6f6f372f6298a" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38275 } }
I20260924 16:09:01.231124 17069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3be9e337e5f94c96a4c6f6f372f6298a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be9e337e5f94c96a4c6f6f372f6298a" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38275 } } }
I20260924 16:09:01.231704 17070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3be9e337e5f94c96a4c6f6f372f6298a. Latest consensus state: current_term: 2 leader_uuid: "3be9e337e5f94c96a4c6f6f372f6298a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be9e337e5f94c96a4c6f6f372f6298a" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38275 } } }
I20260924 16:09:01.232506 17070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [sys.catalog]: This master's current role is: LEADER
I20260924 16:09:01.232316 17069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a [sys.catalog]: This master's current role is: LEADER
I20260924 16:09:01.235780 17078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:09:01.243345 17078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:09:01.245083 17078 catalog_manager.cc:1259] Loaded cluster ID: 182fad9526c94a8a9d91815a4a2f92d6
I20260924 16:09:01.245217 17078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:09:01.250347 17078 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:09:01.252497 17078 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a: Loaded TSK: 0
I20260924 16:09:01.254660 17078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:09:01.257956 17064 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260924 16:09:01.728531 17066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:01.729142 17066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:01.729334 17066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:01.741911 17066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:01.742172 17066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:09:01.742403 17066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:09:01.759881 17066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:40065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.214.212:33379/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:43827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=38295
--tserver_master_addrs=127.3.214.254:38275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:01.762387 17066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:01.764576 17066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:01.810812 17105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:01.813200 17104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:01.813728 17108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:01.814394 17066 server_base.cc:1034] running on GCE node
I20260924 16:09:01.815464 17066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:01.823982 17066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:09:01.827497 17066 hybrid_clock.cc:648] HybridClock initialized: now 1790266141827398 us; error 82 us; skew 500 ppm
I20260924 16:09:01.827968 17066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:01 dist-test-slave-1gc3 krb5kdc[16012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266141, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:01.831899 17066 init.cc:377] Logged in from keytab as kudu/127.3.214.193@KRBTEST.COM (short username kudu)
I20260924 16:09:01.849714 17066 webserver.cc:460] Webserver started at http://127.3.214.193:38295/ using document root <none> and password file <none>
I20260924 16:09:01.850947 17066 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:01.851135 17066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 16:09:01 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266141, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
I20260924 16:09:01.883574 17066 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.006s sys 0.001s
I20260924 16:09:01.887401 17116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:01.889446 17066 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:09:01.889705 17066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "232cdf436bfb45588d65edbd00c4e236"
format_stamp: "Formatted at 2026-09-24 16:08:58 on dist-test-slave-1gc3"
server_key: "46ab8abcf4b42f89c5e69d7e14555322"
server_key_iv: "0edf38c6bfdbfe5accee55090d204317"
server_key_version: "kuduclusterkey@0"
I20260924 16:09:01.891515 17066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:01.913841 17066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:01.919430 17066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:01.919806 17066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:01.921271 17066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:09:01.923588 17066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:09:01.923723 17066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:01.923866 17066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:09:01.923938 17066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:01.987159 17066 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:43827
I20260924 16:09:01.987186 17229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:43827 every 8 connection(s)
I20260924 16:09:01.989419 17066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:09:01.994694 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 17066
I20260924 16:09:01.995621 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.214.212:33379/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:41255
--local_ip_for_outbound_sockets=127.3.214.194
--tserver_master_addrs=127.3.214.254:38275
--webserver_port=38305
--webserver_interface=127.3.214.194
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:40065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.214.212:33379/kms with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:09:02 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266141, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:09:02.056360 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:01.993626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:35631 (local address 127.3.214.254:38275)
0924 16:09:02.020081 (+ 26455us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:02.020109 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:02.020163 (+ 54us) server_negotiation.cc:407] Connection header received
0924 16:09:02.020698 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:02.020721 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:02.021351 (+ 630us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:02.021907 (+ 556us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:02.025526 (+ 3619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:02.027686 (+ 2160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:02.032102 (+ 4416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:02.033496 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:02.033734 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:02.039567 (+ 5833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:02.039585 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:02.039623 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:02.039709 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:02.046466 (+ 6757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:02.047865 (+ 1399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:02.047882 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:02.047896 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:02.048252 (+ 356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:02.048936 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:02.048941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:02.048945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:02.049883 (+ 938us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:02.050177 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:02.054410 (+ 4233us) server_negotiation.cc:299] Negotiation successful
0924 16:09:02.054981 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16034,"thread_start_us":15748,"threads_started":1}
I20260924 16:09:02.057821 17233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:01.993661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38275 (local address 127.3.214.193:35631)
0924 16:09:02.016290 (+ 22629us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:02.016407 (+ 117us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:02.019405 (+ 2998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:02.022838 (+ 3433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:02.022851 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:02.023972 (+ 1121us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:02.025183 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:02.025220 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:02.028050 (+ 2830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:02.028056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:02.031813 (+ 3757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:02.031845 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:02.034349 (+ 2504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:02.034357 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:02.034708 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:02.035637 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:02.035672 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:02.038980 (+ 3308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:02.046780 (+ 7800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:02.046792 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:02.046812 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:02.047556 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:02.048440 (+ 884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:02.048446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:02.048450 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:02.048759 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:02.050311 (+ 1552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:02.050322 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:02.050829 (+ 507us) client_negotiation.cc:769] Sending connection context
0924 16:09:02.051207 (+ 378us) client_negotiation.cc:240] Negotiation successful
0924 16:09:02.051964 (+ 757us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22182,"thread_start_us":20569,"threads_started":1}
I20260924 16:09:02.062983 17230 heartbeater.cc:347] Connected to a master server at 127.3.214.254:38275
I20260924 16:09:02.063398 17230 heartbeater.cc:464] Registering TS with master...
I20260924 16:09:02.064482 17230 heartbeater.cc:511] Master 127.3.214.254:38275 requested a full tablet report, sending...
W20260924 16:09:02.068845 17230 heartbeater.cc:644] Failed to heartbeat to 127.3.214.254:38275 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 16:09:02.069324 17230 heartbeater.cc:464] Registering TS with master...
I20260924 16:09:02.069778 17230 heartbeater.cc:511] Master 127.3.214.254:38275 requested a full tablet report, sending...
I20260924 16:09:02.071432 17230 heartbeater.cc:464] Registering TS with master...
I20260924 16:09:02.071943 17230 heartbeater.cc:511] Master 127.3.214.254:38275 requested a full tablet report, sending...
W20260924 16:09:02.073402 17230 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:09:02.468125 17234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:02.468840 17234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:02.469071 17234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:02.481938 17234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:02.482275 17234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:09:02.482548 17234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:09:02.500569 17234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:40065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.214.212:33379/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:41255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=38305
--tserver_master_addrs=127.3.214.254:38275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:02.503116 17234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:02.505124 17234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:02.544184 17251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:02.546880 17250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:02.550323 17234 server_base.cc:1034] running on GCE node
W20260924 16:09:02.553043 17253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:02.554406 17234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:02.556275 17234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:09:02.557593 17234 hybrid_clock.cc:648] HybridClock initialized: now 1790266142557514 us; error 63 us; skew 500 ppm
I20260924 16:09:02.558139 17234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:02 dist-test-slave-1gc3 krb5kdc[16012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266142, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:02.563638 17234 init.cc:377] Logged in from keytab as kudu/127.3.214.194@KRBTEST.COM (short username kudu)
I20260924 16:09:02.566606 17234 webserver.cc:460] Webserver started at http://127.3.214.194:38305/ using document root <none> and password file <none>
I20260924 16:09:02.567855 17234 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:02.568049 17234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 16:09:02 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266142, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
Sep 24 16:09:02 dist-test-slave-1gc3 krb5kdc[16012](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790266142, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:02.628288 17234 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.005s sys 0.001s
I20260924 16:09:02.635028 17261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:02.637491 17234 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260924 16:09:02.637751 17234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "d51791d775cf4d2db0a1213426a70260"
format_stamp: "Formatted at 2026-09-24 16:08:59 on dist-test-slave-1gc3"
server_key: "b0c4229c739a80f9750b7eb56abeb08b"
server_key_iv: "248a579fe8c79aed9698a5b622220164"
server_key_version: "kuduclusterkey@0"
I20260924 16:09:02.639405 17234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:02.675457 17234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:02.682605 17234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:02.682945 17234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:02.684386 17234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:09:02.686519 17234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:09:02.686671 17234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:02.686812 17234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:09:02.686903 17234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:02.760460 17234 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:41255
I20260924 16:09:02.760504 17376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:41255 every 8 connection(s)
I20260924 16:09:02.762374 17234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:09:02.771886 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 17234
I20260924 16:09:02.772890 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.214.212:33379/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:39365
--local_ip_for_outbound_sockets=127.3.214.195
--tserver_master_addrs=127.3.214.254:38275
--webserver_port=39861
--webserver_interface=127.3.214.195
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:40065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.214.212:33379/kms with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:09:02 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266142, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:09:02.817941 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:02.767077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:40937 (local address 127.3.214.254:38275)
0924 16:09:02.782546 (+ 15469us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:02.782553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:02.786194 (+ 3641us) server_negotiation.cc:407] Connection header received
0924 16:09:02.787962 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:02.787969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:02.788080 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:02.788192 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:02.790885 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:02.792011 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:02.794445 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:02.794999 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:02.795119 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:02.800105 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:02.800112 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:02.800116 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:02.800164 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:02.806660 (+ 6496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:02.808628 (+ 1968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:02.808636 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:02.808639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:02.808832 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:02.809500 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:02.809506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:02.809509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:02.809951 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:02.814207 (+ 4256us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:02.817005 (+ 2798us) server_negotiation.cc:299] Negotiation successful
0924 16:09:02.817448 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15288,"thread_start_us":15470,"threads_started":1}
I20260924 16:09:02.821152 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:02.767959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38275 (local address 127.3.214.194:40937)
0924 16:09:02.783434 (+ 15475us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:02.783512 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:02.787520 (+ 4008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:02.788869 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:02.788882 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:02.789597 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:02.790614 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:02.790650 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:02.792260 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:02.792265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:02.794271 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:02.794296 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:02.795424 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:02.795431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:02.795873 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:02.797022 (+ 1149us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:02.797052 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:02.799790 (+ 2738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:02.807595 (+ 7805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:02.807608 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:02.807645 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:02.808402 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:02.809016 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:02.809021 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:02.809025 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:02.809340 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:02.815234 (+ 5894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:02.815247 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:02.816185 (+ 938us) client_negotiation.cc:769] Sending connection context
0924 16:09:02.818298 (+ 2113us) client_negotiation.cc:240] Negotiation successful
0924 16:09:02.819195 (+ 897us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15010,"thread_start_us":14721,"threads_started":1}
I20260924 16:09:02.823684 17377 heartbeater.cc:347] Connected to a master server at 127.3.214.254:38275
I20260924 16:09:02.824138 17377 heartbeater.cc:464] Registering TS with master...
I20260924 16:09:02.825296 17377 heartbeater.cc:511] Master 127.3.214.254:38275 requested a full tablet report, sending...
I20260924 16:09:02.828012 17029 ts_manager.cc:194] Registered new tserver with Master: d51791d775cf4d2db0a1213426a70260 (127.3.214.194:41255)
I20260924 16:09:02.830581 17029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:40937
I20260924 16:09:03.073601 17230 heartbeater.cc:464] Registering TS with master...
I20260924 16:09:03.074281 17230 heartbeater.cc:511] Master 127.3.214.254:38275 requested a full tablet report, sending...
I20260924 16:09:03.075898 17029 ts_manager.cc:194] Registered new tserver with Master: 232cdf436bfb45588d65edbd00c4e236 (127.3.214.193:43827)
I20260924 16:09:03.077241 17029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:35631
Sep 24 16:09:03 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266142, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
W20260924 16:09:03.351531 17381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:03.352216 17381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:03.352447 17381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:03.364454 17381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:03.364763 17381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:09:03.365017 17381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:09:03.382018 17381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:40065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.214.212:33379/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:39365
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=39861
--tserver_master_addrs=127.3.214.254:38275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:03.384603 17381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:03.386965 17381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:03.420843 17386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:03.424306 17387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:03.426810 17390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:03.431273 17381 server_base.cc:1034] running on GCE node
I20260924 16:09:03.432437 17381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:03.446590 17381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Sep 24 16:09:03 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266142, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
I20260924 16:09:03.450160 17381 hybrid_clock.cc:648] HybridClock initialized: now 1790266143450005 us; error 120 us; skew 500 ppm
I20260924 16:09:03.450793 17381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:03 dist-test-slave-1gc3 krb5kdc[16012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266143, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:03.463270 17381 init.cc:377] Logged in from keytab as kudu/127.3.214.195@KRBTEST.COM (short username kudu)
I20260924 16:09:03.478793 17381 webserver.cc:460] Webserver started at http://127.3.214.195:39861/ using document root <none> and password file <none>
I20260924 16:09:03.480103 17381 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:03.480427 17381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 16:09:03 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266143, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for HTTP/127.3.214.212@KRBTEST.COM
I20260924 16:09:03.525094 17381 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.012s sys 0.004s
I20260924 16:09:03.536298 17398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:03.538776 17381 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.007s sys 0.003s
I20260924 16:09:03.539037 17381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "0a2fc5e20c504b918d988a4a5b9d58a8"
format_stamp: "Formatted at 2026-09-24 16:08:59 on dist-test-slave-1gc3"
server_key: "e1f3ff876aecc95db7fdf5ee048d2334"
server_key_iv: "0649179d37bb3bb21375c0ff30142a41"
server_key_version: "kuduclusterkey@0"
I20260924 16:09:03.540783 17381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:03.586623 17381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:03.593231 17381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:03.593641 17381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:03.596282 17381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:09:03.598592 17381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:09:03.598793 17381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 16:09:03.599773 17381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:09:03.599931 17381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:03.679178 17381 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:39365
I20260924 16:09:03.679201 17510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:39365 every 8 connection(s)
I20260924 16:09:03.681629 17381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:09:03.688980 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 17381
Sep 24 16:09:03 dist-test-slave-1gc3 krb5kdc[16012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266143, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:09:03.751365 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:03.690282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48814 (local address 127.3.214.254:38275)
0924 16:09:03.723329 (+ 33047us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:03.723335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:03.726950 (+ 3615us) server_negotiation.cc:407] Connection header received
0924 16:09:03.729840 (+ 2890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:03.729848 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:03.729962 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:03.730070 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:03.733973 (+ 3903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.735111 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:03.739068 (+ 3957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.739668 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:03.739780 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:03.743980 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:03.743987 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:03.743992 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:03.744040 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:03.747561 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:03.748699 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:03.748706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:03.748709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:03.748914 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:03.749396 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:03.749402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:03.749405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:03.749804 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:03.750071 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:03.750720 (+ 649us) server_negotiation.cc:299] Negotiation successful
0924 16:09:03.750963 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32936,"thread_start_us":30832,"threads_started":1}
I20260924 16:09:03.754338 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:03.688588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:44931 (local address 127.3.214.254:38275)
0924 16:09:03.721080 (+ 32492us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:03.721087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:03.726164 (+ 5077us) server_negotiation.cc:407] Connection header received
0924 16:09:03.733823 (+ 7659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:03.733829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:03.733920 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:03.734024 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:03.737054 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.738366 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:03.740862 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.741507 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:03.741604 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:03.746888 (+ 5284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:03.746896 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:03.746901 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:03.746966 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:03.750303 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:03.751303 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:03.751308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:03.751310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:03.751506 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:03.752021 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:03.752026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:03.752029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:03.752361 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:03.752485 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:03.753681 (+ 1196us) server_negotiation.cc:299] Negotiation successful
0924 16:09:03.753922 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32297,"thread_start_us":34724,"threads_started":1}
I20260924 16:09:03.757077 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:03.687770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38275 (local address 127.3.214.195:44931)
0924 16:09:03.724113 (+ 36343us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:03.724195 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:03.733375 (+ 9180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:03.735571 (+ 2196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:03.735586 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:03.736123 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:03.736808 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:03.736831 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.738580 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:03.738586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:03.740670 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:03.740703 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.741993 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:03.742001 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:03.742375 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:03.743176 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:03.743210 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:03.746476 (+ 3266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:03.750502 (+ 4026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:03.750513 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:03.750534 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:03.751156 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:03.751629 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:03.751634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:03.751637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:03.751889 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:03.752687 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:03.752697 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:03.753213 (+ 516us) client_negotiation.cc:769] Sending connection context
0924 16:09:03.754299 (+ 1086us) client_negotiation.cc:240] Negotiation successful
0924 16:09:03.755208 (+ 909us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35899,"thread_start_us":39049,"threads_started":1}
I20260924 16:09:03.759244 17511 heartbeater.cc:347] Connected to a master server at 127.3.214.254:38275
I20260924 16:09:03.759682 17511 heartbeater.cc:464] Registering TS with master...
I20260924 16:09:03.760664 17511 heartbeater.cc:511] Master 127.3.214.254:38275 requested a full tablet report, sending...
I20260924 16:09:03.764919 17029 ts_manager.cc:194] Registered new tserver with Master: 0a2fc5e20c504b918d988a4a5b9d58a8 (127.3.214.195:39365)
I20260924 16:09:03.766052 17029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:44931
I20260924 16:09:03.769222 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:09:03.787748 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:03.772728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48820 (local address 127.3.214.254:38275)
0924 16:09:03.772967 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:03.772975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:03.773552 (+ 577us) server_negotiation.cc:407] Connection header received
0924 16:09:03.773738 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:03.773742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:03.773812 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:03.773902 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:03.775790 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.776538 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:03.778270 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.778691 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:03.778793 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:03.781687 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:03.781691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:03.781695 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:03.781735 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:03.784398 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:03.785366 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:03.785371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:03.785373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:03.785533 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:03.786091 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:03.786153 (+ 62us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:03.786155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:03.786476 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:03.786687 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:03.787223 (+ 536us) server_negotiation.cc:299] Negotiation successful
0924 16:09:03.787440 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 16:09:03.795037 17029 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:09:03.804849 17029 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:09:03.836963 17377 heartbeater.cc:503] Master 127.3.214.254:38275 was elected leader, sending a full tablet report...
I20260924 16:09:03.839175 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:03.824593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:41255 (local address 127.0.0.1:57726)
0924 16:09:03.826222 (+ 1629us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:03.826243 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:03.826363 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:03.829636 (+ 3273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:03.829640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:03.829661 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:09:03.829939 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:03.829948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.832094 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:03.832098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:03.835020 (+ 2922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:03.835048 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.837876 (+ 2828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:03.837885 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:03.838179 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:03.838255 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 16:09:03.838424 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 16:09:03.838687 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1449,"mutex_wait_us":135,"thread_start_us":1293,"threads_started":1}
I20260924 16:09:03.839658 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:03.823820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:43827 (local address 127.0.0.1:57078)
0924 16:09:03.825324 (+ 1504us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:03.825380 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:03.825565 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:03.830527 (+ 4962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:03.830534 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:03.830574 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:09:03.831068 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:03.831085 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.833535 (+ 2450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:03.833542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:03.835837 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:03.835858 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.838674 (+ 2816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:03.838681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:03.838869 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:03.838905 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 16:09:03.838978 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 16:09:03.839155 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1218,"thread_start_us":672,"threads_started":1}
I20260924 16:09:03.841046 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:03.826556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:39365 (local address 127.0.0.1:37786)
0924 16:09:03.827455 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:03.827476 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:03.827694 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:03.828868 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:03.828875 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:03.828908 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:09:03.829332 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:03.829347 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.831791 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:03.831798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:03.834954 (+ 3156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:03.834980 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.840101 (+ 5121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:03.840107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:03.840311 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:03.840347 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 16:09:03.840435 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 16:09:03.840632 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":81,"thread_start_us":649,"threads_started":1}
I20260924 16:09:03.841305 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:03.824166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57078 (local address 127.3.214.193:43827)
0924 16:09:03.829572 (+ 5406us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:03.829584 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:03.829619 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:09:03.829747 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:03.829756 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:03.830154 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:03.831361 (+ 1207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:09:03.831686 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.833301 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:03.836696 (+ 3395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.838376 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:03.840346 (+ 1970us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:03.840637 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:03.840728 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 16:09:03.840945 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5190,"thread_start_us":4991,"threads_started":1}
I20260924 16:09:03.842140 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:03.826094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.3.214.195:39365)
0924 16:09:03.826964 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:03.826973 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:03.827584 (+ 611us) server_negotiation.cc:407] Connection header received
0924 16:09:03.827897 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:03.827907 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:03.828234 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:03.828438 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:09:03.829654 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.831559 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:03.835237 (+ 3678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.837074 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:03.837234 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:03.839857 (+ 2623us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:03.841465 (+ 1608us) server_negotiation.cc:299] Negotiation successful
0924 16:09:03.841708 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":664,"threads_started":1}
I20260924 16:09:03.842545 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:03.825544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57726 (local address 127.3.214.194:41255)
0924 16:09:03.828957 (+ 3413us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:03.828967 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:03.829003 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:09:03.829119 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:03.829128 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:03.829396 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:03.830122 (+ 726us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:09:03.830395 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.831931 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:03.835256 (+ 3325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.837003 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:03.837163 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:03.837452 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:03.840175 (+ 2723us) server_negotiation.cc:299] Negotiation successful
0924 16:09:03.840409 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3179,"thread_start_us":3003,"threads_started":1}
I20260924 16:09:03.846643 17446 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa8bb6192d04535bebdb4a692529716 (DEFAULT_TABLE table=test-table [id=1d99747b41d0408ba91e0f1201224137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:09:03.847008 17164 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa8bb6192d04535bebdb4a692529716 (DEFAULT_TABLE table=test-table [id=1d99747b41d0408ba91e0f1201224137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:09:03.846819 17310 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa8bb6192d04535bebdb4a692529716 (DEFAULT_TABLE table=test-table [id=1d99747b41d0408ba91e0f1201224137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:09:03.848740 17446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa8bb6192d04535bebdb4a692529716. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:09:03.848742 17164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa8bb6192d04535bebdb4a692529716. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:09:03.848919 17310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa8bb6192d04535bebdb4a692529716. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:09:03.864904 17539 tablet_bootstrap.cc:492] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260: Bootstrap starting.
I20260924 16:09:03.869138 17539 tablet_bootstrap.cc:654] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260: Neither blocks nor log segments found. Creating new log.
I20260924 16:09:03.870090 17541 tablet_bootstrap.cc:492] T 5fa8bb6192d04535bebdb4a692529716 P 0a2fc5e20c504b918d988a4a5b9d58a8: Bootstrap starting.
I20260924 16:09:03.871841 17539 log.cc:824] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260: Log is configured to *not* fsync() on all Append() calls
I20260924 16:09:03.875077 17540 tablet_bootstrap.cc:492] T 5fa8bb6192d04535bebdb4a692529716 P 232cdf436bfb45588d65edbd00c4e236: Bootstrap starting.
I20260924 16:09:03.875113 17541 tablet_bootstrap.cc:654] T 5fa8bb6192d04535bebdb4a692529716 P 0a2fc5e20c504b918d988a4a5b9d58a8: Neither blocks nor log segments found. Creating new log.
I20260924 16:09:03.876797 17541 log.cc:824] T 5fa8bb6192d04535bebdb4a692529716 P 0a2fc5e20c504b918d988a4a5b9d58a8: Log is configured to *not* fsync() on all Append() calls
I20260924 16:09:03.879403 17540 tablet_bootstrap.cc:654] T 5fa8bb6192d04535bebdb4a692529716 P 232cdf436bfb45588d65edbd00c4e236: Neither blocks nor log segments found. Creating new log.
I20260924 16:09:03.880903 17541 tablet_bootstrap.cc:492] T 5fa8bb6192d04535bebdb4a692529716 P 0a2fc5e20c504b918d988a4a5b9d58a8: No bootstrap required, opened a new log
I20260924 16:09:03.880851 17540 log.cc:824] T 5fa8bb6192d04535bebdb4a692529716 P 232cdf436bfb45588d65edbd00c4e236: Log is configured to *not* fsync() on all Append() calls
I20260924 16:09:03.881237 17541 ts_tablet_manager.cc:1397] T 5fa8bb6192d04535bebdb4a692529716 P 0a2fc5e20c504b918d988a4a5b9d58a8: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260924 16:09:03.881448 17539 tablet_bootstrap.cc:492] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260: No bootstrap required, opened a new log
I20260924 16:09:03.881860 17539 ts_tablet_manager.cc:1397] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.000s
I20260924 16:09:03.885156 17540 tablet_bootstrap.cc:492] T 5fa8bb6192d04535bebdb4a692529716 P 232cdf436bfb45588d65edbd00c4e236: No bootstrap required, opened a new log
I20260924 16:09:03.885545 17540 ts_tablet_manager.cc:1397] T 5fa8bb6192d04535bebdb4a692529716 P 232cdf436bfb45588d65edbd00c4e236: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260924 16:09:03.889858 17539 raft_consensus.cc:348] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232cdf436bfb45588d65edbd00c4e236" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43827 } } peers { permanent_uuid: "0a2fc5e20c504b918d988a4a5b9d58a8" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 39365 } } peers { permanent_uuid: "d51791d775cf4d2db0a1213426a70260" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 41255 } }
I20260924 16:09:03.890355 17539 raft_consensus.cc:374] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:09:03.890475 17539 raft_consensus.cc:729] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d51791d775cf4d2db0a1213426a70260, State: Initialized, Role: FOLLOWER
I20260924 16:09:03.891148 17539 consensus_queue.cc:260] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232cdf436bfb45588d65edbd00c4e236" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43827 } } peers { permanent_uuid: "0a2fc5e20c504b918d988a4a5b9d58a8" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 39365 } } peers { permanent_uuid: "d51791d775cf4d2db0a1213426a70260" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 41255 } }
I20260924 16:09:03.893163 17539 ts_tablet_manager.cc:1428] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260: Time spent starting tablet: real 0.011s user 0.013s sys 0.000s
I20260924 16:09:03.893038 17541 raft_consensus.cc:348] T 5fa8bb6192d04535bebdb4a692529716 P 0a2fc5e20c504b918d988a4a5b9d58a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232cdf436bfb45588d65edbd00c4e236" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43827 } } peers { permanent_uuid: "0a2fc5e20c504b918d988a4a5b9d58a8" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 39365 } } peers { permanent_uuid: "d51791d775cf4d2db0a1213426a70260" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 41255 } }
I20260924 16:09:03.893630 17541 raft_consensus.cc:374] T 5fa8bb6192d04535bebdb4a692529716 P 0a2fc5e20c504b918d988a4a5b9d58a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:09:03.893751 17541 raft_consensus.cc:729] T 5fa8bb6192d04535bebdb4a692529716 P 0a2fc5e20c504b918d988a4a5b9d58a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2fc5e20c504b918d988a4a5b9d58a8, State: Initialized, Role: FOLLOWER
I20260924 16:09:03.894596 17541 consensus_queue.cc:260] T 5fa8bb6192d04535bebdb4a692529716 P 0a2fc5e20c504b918d988a4a5b9d58a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232cdf436bfb45588d65edbd00c4e236" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43827 } } peers { permanent_uuid: "0a2fc5e20c504b918d988a4a5b9d58a8" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 39365 } } peers { permanent_uuid: "d51791d775cf4d2db0a1213426a70260" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 41255 } }
I20260924 16:09:03.896838 17541 ts_tablet_manager.cc:1428] T 5fa8bb6192d04535bebdb4a692529716 P 0a2fc5e20c504b918d988a4a5b9d58a8: Time spent starting tablet: real 0.015s user 0.017s sys 0.000s
I20260924 16:09:03.897229 17511 heartbeater.cc:503] Master 127.3.214.254:38275 was elected leader, sending a full tablet report...
I20260924 16:09:03.897060 17540 raft_consensus.cc:348] T 5fa8bb6192d04535bebdb4a692529716 P 232cdf436bfb45588d65edbd00c4e236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232cdf436bfb45588d65edbd00c4e236" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43827 } } peers { permanent_uuid: "0a2fc5e20c504b918d988a4a5b9d58a8" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 39365 } } peers { permanent_uuid: "d51791d775cf4d2db0a1213426a70260" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 41255 } }
I20260924 16:09:03.897611 17540 raft_consensus.cc:374] T 5fa8bb6192d04535bebdb4a692529716 P 232cdf436bfb45588d65edbd00c4e236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:09:03.897715 17540 raft_consensus.cc:729] T 5fa8bb6192d04535bebdb4a692529716 P 232cdf436bfb45588d65edbd00c4e236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 232cdf436bfb45588d65edbd00c4e236, State: Initialized, Role: FOLLOWER
I20260924 16:09:03.898501 17540 consensus_queue.cc:260] T 5fa8bb6192d04535bebdb4a692529716 P 232cdf436bfb45588d65edbd00c4e236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232cdf436bfb45588d65edbd00c4e236" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43827 } } peers { permanent_uuid: "0a2fc5e20c504b918d988a4a5b9d58a8" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 39365 } } peers { permanent_uuid: "d51791d775cf4d2db0a1213426a70260" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 41255 } }
I20260924 16:09:03.899916 17230 heartbeater.cc:503] Master 127.3.214.254:38275 was elected leader, sending a full tablet report...
I20260924 16:09:03.900739 17540 ts_tablet_manager.cc:1428] T 5fa8bb6192d04535bebdb4a692529716 P 232cdf436bfb45588d65edbd00c4e236: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260924 16:09:03.983820 17512 tablet.cc:2369] T 5fa8bb6192d04535bebdb4a692529716 P 0a2fc5e20c504b918d988a4a5b9d58a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:09:03.992120 17545 raft_consensus.cc:482] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:09:03.992601 17545 raft_consensus.cc:504] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232cdf436bfb45588d65edbd00c4e236" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43827 } } peers { permanent_uuid: "0a2fc5e20c504b918d988a4a5b9d58a8" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 39365 } } peers { permanent_uuid: "d51791d775cf4d2db0a1213426a70260" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 41255 } }
W20260924 16:09:03.993391 17231 tablet.cc:2369] T 5fa8bb6192d04535bebdb4a692529716 P 232cdf436bfb45588d65edbd00c4e236: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:09:03.995044 17545 leader_election.cc:288] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 232cdf436bfb45588d65edbd00c4e236 (127.3.214.193:43827), 0a2fc5e20c504b918d988a4a5b9d58a8 (127.3.214.195:39365)
I20260924 16:09:04.003185 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:03.995648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:54229 (local address 127.3.214.193:43827)
0924 16:09:03.995866 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:03.995871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:03.996255 (+ 384us) server_negotiation.cc:407] Connection header received
0924 16:09:03.996533 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:03.996539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:03.996638 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:03.996744 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:09:03.997502 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.998467 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:04.000846 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:04.002048 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:04.002226 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:04.002287 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:04.002803 (+ 516us) server_negotiation.cc:299] Negotiation successful
0924 16:09:04.002916 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:09:04.003494 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:03.995360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:43827 (local address 127.3.214.194:54229)
0924 16:09:03.996098 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:03.996123 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:03.996312 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:03.996873 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:03.996881 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:03.996921 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:09:03.997345 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:03.997363 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.998639 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:03.998647 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:04.000664 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:04.000681 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:04.002327 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:04.002335 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:04.002560 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:04.002609 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 16:09:04.002746 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 16:09:04.002961 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":1,"thread_start_us":362,"threads_started":1}
I20260924 16:09:04.004637 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:03.996411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:45187 (local address 127.3.214.195:39365)
0924 16:09:03.996634 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:03.996640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:03.997155 (+ 515us) server_negotiation.cc:407] Connection header received
0924 16:09:03.997363 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:03.997369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:03.997449 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:03.997544 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:09:03.998211 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.999317 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:04.002046 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:04.003466 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:04.003679 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:04.003778 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:04.004179 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 16:09:04.004371 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 16:09:04.004628 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:03.996219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:39365 (local address 127.3.214.194:45187)
0924 16:09:03.997017 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:03.997034 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:03.997199 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:03.997574 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:03.997579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:03.997604 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:09:03.998019 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:03.998035 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:03.999601 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:03.999607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:04.001802 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:04.001825 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:04.003796 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:04.003802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:04.003958 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:04.003990 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 16:09:04.004089 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 16:09:04.004252 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":411,"threads_started":1}
I20260924 16:09:04.005188 17184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa8bb6192d04535bebdb4a692529716" candidate_uuid: "d51791d775cf4d2db0a1213426a70260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "232cdf436bfb45588d65edbd00c4e236" is_pre_election: true
I20260924 16:09:04.005869 17184 raft_consensus.cc:2454] T 5fa8bb6192d04535bebdb4a692529716 P 232cdf436bfb45588d65edbd00c4e236 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d51791d775cf4d2db0a1213426a70260 in term 0.
I20260924 16:09:04.005827 17466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa8bb6192d04535bebdb4a692529716" candidate_uuid: "d51791d775cf4d2db0a1213426a70260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fc5e20c504b918d988a4a5b9d58a8" is_pre_election: true
I20260924 16:09:04.006397 17466 raft_consensus.cc:2454] T 5fa8bb6192d04535bebdb4a692529716 P 0a2fc5e20c504b918d988a4a5b9d58a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d51791d775cf4d2db0a1213426a70260 in term 0.
I20260924 16:09:04.006770 17264 leader_election.cc:302] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 232cdf436bfb45588d65edbd00c4e236, d51791d775cf4d2db0a1213426a70260; no voters:
I20260924 16:09:04.007292 17545 raft_consensus.cc:2781] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:09:04.007454 17545 raft_consensus.cc:482] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:09:04.007575 17545 raft_consensus.cc:3037] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:09:04.009994 17545 raft_consensus.cc:504] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232cdf436bfb45588d65edbd00c4e236" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43827 } } peers { permanent_uuid: "0a2fc5e20c504b918d988a4a5b9d58a8" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 39365 } } peers { permanent_uuid: "d51791d775cf4d2db0a1213426a70260" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 41255 } }
I20260924 16:09:04.010761 17545 leader_election.cc:288] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260 [CANDIDATE]: Term 1 election: Requested vote from peers 232cdf436bfb45588d65edbd00c4e236 (127.3.214.193:43827), 0a2fc5e20c504b918d988a4a5b9d58a8 (127.3.214.195:39365)
I20260924 16:09:04.011488 17466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa8bb6192d04535bebdb4a692529716" candidate_uuid: "d51791d775cf4d2db0a1213426a70260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2fc5e20c504b918d988a4a5b9d58a8"
I20260924 16:09:04.011472 17184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa8bb6192d04535bebdb4a692529716" candidate_uuid: "d51791d775cf4d2db0a1213426a70260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "232cdf436bfb45588d65edbd00c4e236"
I20260924 16:09:04.011708 17466 raft_consensus.cc:3037] T 5fa8bb6192d04535bebdb4a692529716 P 0a2fc5e20c504b918d988a4a5b9d58a8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:09:04.011739 17184 raft_consensus.cc:3037] T 5fa8bb6192d04535bebdb4a692529716 P 232cdf436bfb45588d65edbd00c4e236 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:09:04.014497 17184 raft_consensus.cc:2454] T 5fa8bb6192d04535bebdb4a692529716 P 232cdf436bfb45588d65edbd00c4e236 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d51791d775cf4d2db0a1213426a70260 in term 1.
I20260924 16:09:04.014995 17466 raft_consensus.cc:2454] T 5fa8bb6192d04535bebdb4a692529716 P 0a2fc5e20c504b918d988a4a5b9d58a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d51791d775cf4d2db0a1213426a70260 in term 1.
I20260924 16:09:04.015038 17264 leader_election.cc:302] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 232cdf436bfb45588d65edbd00c4e236, d51791d775cf4d2db0a1213426a70260; no voters:
I20260924 16:09:04.015393 17545 raft_consensus.cc:2781] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:09:04.016013 17545 raft_consensus.cc:686] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260 [term 1 LEADER]: Becoming Leader. State: Replica: d51791d775cf4d2db0a1213426a70260, State: Running, Role: LEADER
I20260924 16:09:04.016623 17545 consensus_queue.cc:237] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232cdf436bfb45588d65edbd00c4e236" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43827 } } peers { permanent_uuid: "0a2fc5e20c504b918d988a4a5b9d58a8" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 39365 } } peers { permanent_uuid: "d51791d775cf4d2db0a1213426a70260" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 41255 } }
I20260924 16:09:04.022948 17028 catalog_manager.cc:5495] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260 reported cstate change: term changed from 0 to 1, leader changed from <none> to d51791d775cf4d2db0a1213426a70260 (127.3.214.194). New cstate: current_term: 1 leader_uuid: "d51791d775cf4d2db0a1213426a70260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232cdf436bfb45588d65edbd00c4e236" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2fc5e20c504b918d988a4a5b9d58a8" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 39365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d51791d775cf4d2db0a1213426a70260" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 41255 } health_report { overall_health: HEALTHY } } }
W20260924 16:09:04.038237 17378 tablet.cc:2369] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:09:04.045773 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:04.039612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57736 (local address 127.3.214.194:41255)
0924 16:09:04.039870 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:04.039876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:04.039899 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:09:04.040064 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:04.040069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:04.040272 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:04.040471 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:09:04.040898 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:04.041666 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:04.043218 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:04.043659 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:04.043780 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:04.044486 (+ 706us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:09:04.045105 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:09:04.045320 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:04.045386 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 16:09:04.045484 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 16:09:04.056770 17466 raft_consensus.cc:1261] T 5fa8bb6192d04535bebdb4a692529716 P 0a2fc5e20c504b918d988a4a5b9d58a8 [term 1 FOLLOWER]: Refusing update from remote peer d51791d775cf4d2db0a1213426a70260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:09:04.056852 17184 raft_consensus.cc:1261] T 5fa8bb6192d04535bebdb4a692529716 P 232cdf436bfb45588d65edbd00c4e236 [term 1 FOLLOWER]: Refusing update from remote peer d51791d775cf4d2db0a1213426a70260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:09:04.058076 17545 consensus_queue.cc:1035] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "232cdf436bfb45588d65edbd00c4e236" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:09:04.058534 17550 consensus_queue.cc:1035] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2fc5e20c504b918d988a4a5b9d58a8" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 39365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:09:04.092013 17028 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:09:04.092434 17028 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:09:04.097756 17028 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3be9e337e5f94c96a4c6f6f372f6298a: Sending DeleteTablet for 3 replicas of tablet 5fa8bb6192d04535bebdb4a692529716
I20260924 16:09:04.099653 17446 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fa8bb6192d04535bebdb4a692529716 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:09:04 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:37786
I20260924 16:09:04.099677 17310 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fa8bb6192d04535bebdb4a692529716 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:09:04 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:57726
I20260924 16:09:04.099608 17164 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fa8bb6192d04535bebdb4a692529716 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:09:04 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:57078
I20260924 16:09:04.101231 17564 tablet_replica.cc:330] T 5fa8bb6192d04535bebdb4a692529716 P 232cdf436bfb45588d65edbd00c4e236: stopping tablet replica
I20260924 16:09:04.101940 17564 raft_consensus.cc:2229] T 5fa8bb6192d04535bebdb4a692529716 P 232cdf436bfb45588d65edbd00c4e236 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:09:04.102747 17564 raft_consensus.cc:2258] T 5fa8bb6192d04535bebdb4a692529716 P 232cdf436bfb45588d65edbd00c4e236 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:09:04.103746 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 17066
I20260924 16:09:04.112502 17563 tablet_replica.cc:330] T 5fa8bb6192d04535bebdb4a692529716 P 0a2fc5e20c504b918d988a4a5b9d58a8: stopping tablet replica
I20260924 16:09:04.113159 17563 raft_consensus.cc:2229] T 5fa8bb6192d04535bebdb4a692529716 P 0a2fc5e20c504b918d988a4a5b9d58a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:09:04.113848 17563 raft_consensus.cc:2258] T 5fa8bb6192d04535bebdb4a692529716 P 0a2fc5e20c504b918d988a4a5b9d58a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:09:04.116293 17562 tablet_replica.cc:330] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260: stopping tablet replica
I20260924 16:09:04.116518 17563 ts_tablet_manager.cc:1905] T 5fa8bb6192d04535bebdb4a692529716 P 0a2fc5e20c504b918d988a4a5b9d58a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:09:04.117157 17562 raft_consensus.cc:2229] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:09:04.118309 17562 raft_consensus.cc:2258] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:09:04.121404 17564 ts_tablet_manager.cc:1905] T 5fa8bb6192d04535bebdb4a692529716 P 232cdf436bfb45588d65edbd00c4e236: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:09:04.124732 17563 ts_tablet_manager.cc:1918] T 5fa8bb6192d04535bebdb4a692529716 P 0a2fc5e20c504b918d988a4a5b9d58a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:09:04.124914 17563 log.cc:1196] T 5fa8bb6192d04535bebdb4a692529716 P 0a2fc5e20c504b918d988a4a5b9d58a8: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/5fa8bb6192d04535bebdb4a692529716
I20260924 16:09:04.125438 17563 ts_tablet_manager.cc:1939] T 5fa8bb6192d04535bebdb4a692529716 P 0a2fc5e20c504b918d988a4a5b9d58a8: Deleting consensus metadata
I20260924 16:09:04.127655 17015 catalog_manager.cc:4841] TS 0a2fc5e20c504b918d988a4a5b9d58a8 (127.3.214.195:39365): tablet 5fa8bb6192d04535bebdb4a692529716 (table test-table [id=1d99747b41d0408ba91e0f1201224137]) successfully deleted
I20260924 16:09:04.130154 17562 ts_tablet_manager.cc:1905] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:09:04.130514 17564 ts_tablet_manager.cc:1918] T 5fa8bb6192d04535bebdb4a692529716 P 232cdf436bfb45588d65edbd00c4e236: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:09:04.130753 17564 log.cc:1196] T 5fa8bb6192d04535bebdb4a692529716 P 232cdf436bfb45588d65edbd00c4e236: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-0/wal/wals/5fa8bb6192d04535bebdb4a692529716
I20260924 16:09:04.131204 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:04.106759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57080 (local address 127.3.214.193:43827)
0924 16:09:04.106971 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:04.106978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:04.106999 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:09:04.107068 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:04.107075 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:04.107164 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:04.107266 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:04.109834 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:04.111112 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:04.114640 (+ 3528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:04.115260 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:04.115395 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:04.119947 (+ 4552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:04.119969 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:04.119990 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:04.120041 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:04.124989 (+ 4948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:04.126086 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:04.126116 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:04.126120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:04.126291 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:04.126801 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:04.126806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:04.126810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:04.127384 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:04.127774 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:04.130452 (+ 2678us) server_negotiation.cc:299] Negotiation successful
0924 16:09:04.130763 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 16:09:04.131476 17564 ts_tablet_manager.cc:1939] T 5fa8bb6192d04535bebdb4a692529716 P 232cdf436bfb45588d65edbd00c4e236: Deleting consensus metadata
I20260924 16:09:04.132225 17225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:04.141696 17562 ts_tablet_manager.cc:1918] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:09:04.141971 17562 log.cc:1196] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/5fa8bb6192d04535bebdb4a692529716
I20260924 16:09:04.142735 17562 ts_tablet_manager.cc:1939] T 5fa8bb6192d04535bebdb4a692529716 P d51791d775cf4d2db0a1213426a70260: Deleting consensus metadata
I20260924 16:09:04.145207 17013 catalog_manager.cc:4841] TS d51791d775cf4d2db0a1213426a70260 (127.3.214.194:41255): tablet 5fa8bb6192d04535bebdb4a692529716 (table test-table [id=1d99747b41d0408ba91e0f1201224137]) successfully deleted
I20260924 16:09:04.242039 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 17066
I20260924 16:09:04.242063 17015 catalog_manager.cc:4841] TS 232cdf436bfb45588d65edbd00c4e236 (127.3.214.193:43827): tablet 5fa8bb6192d04535bebdb4a692529716 (table test-table [id=1d99747b41d0408ba91e0f1201224137]) successfully deleted
I20260924 16:09:04.271855 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 17234
I20260924 16:09:04.286629 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:04.272917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.3.214.194:41255)
0924 16:09:04.273102 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:04.273108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:04.273239 (+ 131us) server_negotiation.cc:407] Connection header received
0924 16:09:04.273442 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:04.273449 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:04.273525 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:04.273623 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:04.275020 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:04.275938 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:04.277140 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:04.277592 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:04.277714 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:04.280414 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:04.280434 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:04.280457 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:04.280493 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:04.283710 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:04.284426 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:04.284437 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:04.284440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:04.284568 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:04.284929 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:04.284933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:04.284935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:04.285326 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:04.285672 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:04.285945 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 16:09:04.286260 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 16:09:04.287343 17372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:04.418670 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 17234
I20260924 16:09:04.437081 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 17381
I20260924 16:09:04.454540 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:04.438073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37798 (local address 127.3.214.195:39365)
0924 16:09:04.438332 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:04.438338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:04.438363 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:09:04.438560 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:04.438572 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:04.438673 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:04.438796 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:04.440113 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:04.440963 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:04.442553 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:04.443038 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:04.443156 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:04.446082 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:04.446148 (+ 66us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:04.446181 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:04.446262 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:04.450360 (+ 4098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:04.451514 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:04.451529 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:04.451534 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:04.451779 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:04.452283 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:04.452288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:04.452292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:04.452941 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:04.453352 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:04.453781 (+ 429us) server_negotiation.cc:299] Negotiation successful
0924 16:09:04.454058 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260924 16:09:04.455448 17506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:04.553560 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 17381
I20260924 16:09:04.569972 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 16996
I20260924 16:09:04.571456 17059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:04.695351 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 16996
I20260924 16:09:04.725311 3931 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 16:09:05.072999 3931 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 16:09:05.073289 3931 mini_ranger.cc:66] Stopping Ranger...
I20260924 16:09:05.420835 3931 mini_ranger.cc:68] Stopped Ranger
2026-09-24 16:09:05.421 UTC [16042] LOG: received smart shutdown request
2026-09-24 16:09:05.424 UTC [16042] LOG: background worker "logical replication launcher" (PID 16051) exited with exit code 1
2026-09-24 16:09:05.424 UTC [16046] LOG: shutting down
2026-09-24 16:09:05.604 UTC [16042] LOG: database system is shut down
2026-09-24T16:09:05Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (57601 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:09:05 dist-test-slave-1gc3 krb5kdc[17578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:09:05 dist-test-slave-1gc3 krb5kdc[17578](info): set up 2 sockets
Sep 24 16:09:05 dist-test-slave-1gc3 krb5kdc[17578](info): commencing operation
krb5kdc: starting...
W20260924 16:09:06.771852 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:09:06 dist-test-slave-1gc3 krb5kdc[17578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:09:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:09:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:09:07.058597 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:35729
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:42019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:35729,127.3.214.253:41989,127.3.214.252:43419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:09:07.463055 17591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:07.463465 17591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:07.463630 17591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:07.471975 17591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:09:07.472136 17591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:07.472191 17591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:09:07.472282 17591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:09:07.472354 17591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:09:07.484102 17591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:42019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:35729,127.3.214.253:41989,127.3.214.252:43419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:35729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:07.486024 17591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:07.487789 17591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:07.502190 17596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:07.502170 17599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:07.503027 17597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:07.503362 17591 server_base.cc:1034] running on GCE node
I20260924 16:09:07.504381 17591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:07.506287 17591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:09:07.507529 17591 hybrid_clock.cc:648] HybridClock initialized: now 1790266147507446 us; error 67 us; skew 500 ppm
I20260924 16:09:07.507949 17591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:07 dist-test-slave-1gc3 krb5kdc[17578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266147, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:07.519323 17591 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:09:07.521219 17591 webserver.cc:460] Webserver started at http://127.3.214.254:42571/ using document root <none> and password file <none>
I20260924 16:09:07.522024 17591 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:07.522194 17591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:09:07.522509 17591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:09:07.525826 17591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "3f3dd35ee12f44bc93251d1c7d2fe357"
format_stamp: "Formatted at 2026-09-24 16:09:07 on dist-test-slave-1gc3"
server_key: "c781629b13f083ff661174961dd35796"
server_key_iv: "3d21fb332b8a0af8199d0dd168ed3ba1"
server_key_version: "encryptionkey@0"
I20260924 16:09:07.526768 17591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "3f3dd35ee12f44bc93251d1c7d2fe357"
format_stamp: "Formatted at 2026-09-24 16:09:07 on dist-test-slave-1gc3"
server_key: "c781629b13f083ff661174961dd35796"
server_key_iv: "3d21fb332b8a0af8199d0dd168ed3ba1"
server_key_version: "encryptionkey@0"
I20260924 16:09:07.533124 17591 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260924 16:09:07.537638 17606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:07.539634 17591 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260924 16:09:07.539861 17591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "3f3dd35ee12f44bc93251d1c7d2fe357"
format_stamp: "Formatted at 2026-09-24 16:09:07 on dist-test-slave-1gc3"
server_key: "c781629b13f083ff661174961dd35796"
server_key_iv: "3d21fb332b8a0af8199d0dd168ed3ba1"
server_key_version: "encryptionkey@0"
I20260924 16:09:07.540091 17591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:07.568846 17591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:07.573534 17591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:07.573796 17591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:07.595319 17591 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:35729
I20260924 16:09:07.595404 17657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:35729 every 8 connection(s)
I20260924 16:09:07.597054 17591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:09:07.598008 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 17591
I20260924 16:09:07.598277 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:09:07.598922 3931 external_mini_cluster.cc:1428] Setting key edab48b139daa9d54c3b5ebc37f97dbc
I20260924 16:09:07.602187 17658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:09:07.611439 17658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35729 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.214.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.253@KRBTEST.COM" created.
Sep 24 16:09:07 dist-test-slave-1gc3 krb5kdc[17578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266147, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:09:07.641335 17662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:07.616306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39208 (local address 127.3.214.254:35729)
0924 16:09:07.618731 (+ 2425us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:07.618738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:07.618799 (+ 61us) server_negotiation.cc:407] Connection header received
0924 16:09:07.620908 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:07.620915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:07.621232 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:07.621436 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:07.623437 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:07.624720 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:07.627192 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:07.627911 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:07.628064 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:07.632089 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:07.632101 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:07.632134 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:07.632186 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:07.636075 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:07.637021 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:07.637032 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:07.637035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:07.637320 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:07.637766 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:07.637770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:07.637773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:07.638564 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:07.638798 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:07.639536 (+ 738us) server_negotiation.cc:299] Negotiation successful
0924 16:09:07.640055 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":118,"server-negotiator.queue_time_us":2182,"spinlock_wait_cycles":36224,"thread_start_us":1833,"threads_started":1}
I20260924 16:09:07.641404 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:07.616306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:35729 (local address 127.0.0.1:39208)
0924 16:09:07.617532 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:07.617605 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:07.620353 (+ 2748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:07.621555 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:07.621564 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:07.622526 (+ 962us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:07.623148 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:07.623183 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:07.624925 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:07.624933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:07.626991 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:07.627010 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:07.628200 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:07.628206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:07.628385 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:07.629069 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:07.629098 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:07.631733 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:07.636285 (+ 4552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:07.636299 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:07.636314 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:07.636828 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:07.637451 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:07.637455 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:07.637458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:07.637647 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:07.638725 (+ 1078us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:07.638734 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:07.639010 (+ 276us) client_negotiation.cc:769] Sending connection context
0924 16:09:07.639271 (+ 261us) client_negotiation.cc:240] Negotiation successful
0924 16:09:07.639809 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"thread_start_us":557,"threads_started":1}
I20260924 16:09:07.645124 17658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:09:07.646606 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:07.645911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:41989 (local address 127.0.0.1:60042)
0924 16:09:07.646204 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:07.646380 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:41989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260924 16:09:07.647176 17607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.214.253:41989: connect: Connection refused (error 111)
I20260924 16:09:07.649021 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:07.648539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:41989 (local address 127.0.0.1:60044)
0924 16:09:07.648719 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:07.648836 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:41989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 16:09:07.649313 17658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.253:41989: Network error: Client connection negotiation failed: client connection to 127.3.214.253:41989: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.253.keytab.
Entry for principal kudu/127.3.214.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.253.keytab.
I20260924 16:09:07.666762 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.253:41989
--webserver_interface=127.3.214.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.253.keytab
--principal=kudu/127.3.214.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:42019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:35729,127.3.214.253:41989,127.3.214.252:43419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:09:07.698741 17658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } attempt: 1
I20260924 16:09:07.700520 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:07.699798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:41989 (local address 127.0.0.1:60060)
0924 16:09:07.700131 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:07.700308 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:41989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 16:09:07.701810 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:07.701307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:41989 (local address 127.0.0.1:60068)
0924 16:09:07.701511 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:07.701622 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:41989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260924 16:09:07.702036 17658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.253:41989: Network error: Client connection negotiation failed: client connection to 127.3.214.253:41989: connect: Connection refused (error 111)
I20260924 16:09:07.770406 17658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } attempt: 2
I20260924 16:09:07.772220 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:07.771485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:41989 (local address 127.0.0.1:60072)
0924 16:09:07.771835 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:07.771988 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:41989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260924 16:09:07.773494 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:07.773024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:41989 (local address 127.0.0.1:60088)
0924 16:09:07.773185 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:07.773305 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:41989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 16:09:07.773768 17658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.253:41989: Network error: Client connection negotiation failed: client connection to 127.3.214.253:41989: connect: Connection refused (error 111)
I20260924 16:09:07.865149 17658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } attempt: 3
I20260924 16:09:07.866951 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:07.866222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:41989 (local address 127.0.0.1:60096)
0924 16:09:07.866526 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:07.866697 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:41989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 16:09:07.868517 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:07.867944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:41989 (local address 127.0.0.1:60112)
0924 16:09:07.868145 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:07.868290 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:41989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 16:09:07.868837 17658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.253:41989: Network error: Client connection negotiation failed: client connection to 127.3.214.253:41989: connect: Connection refused (error 111)
I20260924 16:09:08.012173 17658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } attempt: 4
I20260924 16:09:08.013890 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.013027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:41989 (local address 127.0.0.1:60124)
0924 16:09:08.013468 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.013660 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:41989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244}
I20260924 16:09:08.015473 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.014896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:41989 (local address 127.0.0.1:60130)
0924 16:09:08.015103 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.015251 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:41989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260924 16:09:08.015771 17658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.253:41989: Network error: Client connection negotiation failed: client connection to 127.3.214.253:41989: connect: Connection refused (error 111)
W20260924 16:09:08.076731 17666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:08.077147 17666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:08.077250 17666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:08.085455 17666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:09:08.085618 17666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:08.085677 17666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:09:08.085745 17666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:09:08.085796 17666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:09:08.097311 17666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:42019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:35729,127.3.214.253:41989,127.3.214.252:43419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.253.keytab
--principal=kudu/127.3.214.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.253:41989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.214.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:08.099207 17666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:08.100903 17666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:08.115056 17674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:08.115123 17672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:08.115056 17671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:08.116223 17666 server_base.cc:1034] running on GCE node
I20260924 16:09:08.117291 17666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:08.119410 17666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:09:08.120649 17666 hybrid_clock.cc:648] HybridClock initialized: now 1790266148120568 us; error 77 us; skew 500 ppm
I20260924 16:09:08.121093 17666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:08 dist-test-slave-1gc3 krb5kdc[17578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266148, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:08.125111 17666 init.cc:377] Logged in from keytab as kudu/127.3.214.253@KRBTEST.COM (short username kudu)
I20260924 16:09:08.127465 17666 webserver.cc:460] Webserver started at http://127.3.214.253:39503/ using document root <none> and password file <none>
I20260924 16:09:08.128479 17666 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:08.128639 17666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:09:08.129047 17666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:09:08.133275 17666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-1/data/instance:
uuid: "43b5a38ca2004e1ca8fd5291880e53de"
format_stamp: "Formatted at 2026-09-24 16:09:08 on dist-test-slave-1gc3"
server_key: "7e2d99bed94072e8bb05a546b1b92e32"
server_key_iv: "c196195981d725d5e76b1467ac570337"
server_key_version: "encryptionkey@0"
I20260924 16:09:08.134467 17666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-1/wal/instance:
uuid: "43b5a38ca2004e1ca8fd5291880e53de"
format_stamp: "Formatted at 2026-09-24 16:09:08 on dist-test-slave-1gc3"
server_key: "7e2d99bed94072e8bb05a546b1b92e32"
server_key_iv: "c196195981d725d5e76b1467ac570337"
server_key_version: "encryptionkey@0"
I20260924 16:09:08.141795 17666 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260924 16:09:08.146991 17681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:08.149056 17666 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:09:08.149386 17666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-1/wal
uuid: "43b5a38ca2004e1ca8fd5291880e53de"
format_stamp: "Formatted at 2026-09-24 16:09:08 on dist-test-slave-1gc3"
server_key: "7e2d99bed94072e8bb05a546b1b92e32"
server_key_iv: "c196195981d725d5e76b1467ac570337"
server_key_version: "encryptionkey@0"
I20260924 16:09:08.149680 17666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:08.178766 17666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:08.184070 17666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:08.184404 17666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:08.207422 17666 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.253:41989
I20260924 16:09:08.207511 17732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.253:41989 every 8 connection(s)
I20260924 16:09:08.209384 17666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-1/data/info.pb
I20260924 16:09:08.214742 17733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:09:08.218446 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 17666
I20260924 16:09:08.218693 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-1/wal/instance
I20260924 16:09:08.219260 3931 external_mini_cluster.cc:1428] Setting key 5407b394f36a58c2912f8f6c9b930418
I20260924 16:09:08.225899 17733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35729 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:09:08 dist-test-slave-1gc3 krb5kdc[17578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266148, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.253@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.214.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.252@KRBTEST.COM" created.
I20260924 16:09:08.254750 17735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.230381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39214 (local address 127.3.214.254:35729)
0924 16:09:08.231036 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:08.231044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:08.232585 (+ 1541us) server_negotiation.cc:407] Connection header received
0924 16:09:08.234889 (+ 2304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:08.234896 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:08.235018 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:08.235190 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:08.237259 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.238490 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.240918 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.241626 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.241825 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:08.246290 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:08.246300 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:08.246307 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:08.246409 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:08.250590 (+ 4181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.251773 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.251779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.251781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.251952 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.252412 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.252417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.252419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.252773 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:08.252925 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:08.254063 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0924 16:09:08.254419 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":300,"threads_started":1}
I20260924 16:09:08.256225 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.231299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:35729 (local address 127.0.0.1:39214)
0924 16:09:08.232364 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.232430 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:08.234464 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:08.235540 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:08.235551 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:08.236246 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:08.236987 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:08.237012 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.238866 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.238873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:08.240585 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:08.240617 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.241965 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.241973 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:08.242307 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:08.243084 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:08.243113 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:08.245831 (+ 2718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:08.250890 (+ 5059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:08.250901 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:08.250921 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:08.251616 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:08.252063 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:08.252066 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:08.252069 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:08.252282 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:08.252951 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:08.252961 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:08.253469 (+ 508us) client_negotiation.cc:769] Sending connection context
0924 16:09:08.253955 (+ 486us) client_negotiation.cc:240] Negotiation successful
0924 16:09:08.254621 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":422,"threads_started":1}
I20260924 16:09:08.258769 17733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:09:08 dist-test-slave-1gc3 krb5kdc[17578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266148, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.253@KRBTEST.COM for kudu/127.3.214.253@KRBTEST.COM
I20260924 16:09:08.278935 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.259786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:41989 (local address 127.0.0.1:60140)
0924 16:09:08.260044 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.260067 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:08.260220 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:08.261748 (+ 1528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:08.261754 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:08.262314 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:08.262539 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:08.262552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.264191 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.264197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:08.265511 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:08.265528 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.266708 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.266726 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:08.266933 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:08.267666 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:08.267689 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:08.270544 (+ 2855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:08.274810 (+ 4266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:08.274819 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:08.274823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:08.275537 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:08.276109 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:08.276114 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:08.276118 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:08.276388 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:08.277305 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:08.277311 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:08.277531 (+ 220us) client_negotiation.cc:769] Sending connection context
0924 16:09:08.277727 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 16:09:08.278139 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 16:09:08.279083 17738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.260107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60140 (local address 127.3.214.253:41989)
0924 16:09:08.261011 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:08.261020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:08.261062 (+ 42us) server_negotiation.cc:407] Connection header received
0924 16:09:08.261134 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:08.261143 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:08.261491 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:08.261677 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:08.262742 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.264021 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.265711 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.266523 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.266641 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:08.270820 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:08.270838 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:08.270862 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:08.270902 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:08.274502 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.275777 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.275788 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.275790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.275965 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.276571 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.276576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.276578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.277120 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:08.277376 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:08.277938 (+ 562us) server_negotiation.cc:299] Negotiation successful
0924 16:09:08.278368 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":670,"thread_start_us":497,"threads_started":1}
I20260924 16:09:08.282449 17733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:09:08.283928 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.283337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:43419 (local address 127.0.0.1:47040)
0924 16:09:08.283545 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.283748 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260924 16:09:08.284281 17682 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
I20260924 16:09:08.286319 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.285820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:43419 (local address 127.0.0.1:47050)
0924 16:09:08.286016 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.286166 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 16:09:08.286615 17733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:43419: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.252.keytab.
Entry for principal kudu/127.3.214.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.252.keytab.
I20260924 16:09:08.290777 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.252:43419
--webserver_interface=127.3.214.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.252.keytab
--principal=kudu/127.3.214.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:42019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:35729,127.3.214.253:41989,127.3.214.252:43419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:09:08.315079 17658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } attempt: 5
Sep 24 16:09:08 dist-test-slave-1gc3 krb5kdc[17578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266147, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for kudu/127.3.214.253@KRBTEST.COM
I20260924 16:09:08.332474 17738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.316343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60146 (local address 127.3.214.253:41989)
0924 16:09:08.316595 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:08.316600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:08.316624 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:09:08.316699 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:08.316703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:08.316782 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:08.316887 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:08.317991 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.318934 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.320869 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.321479 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.321612 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:08.325839 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:08.325847 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:08.325851 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:08.325903 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:08.329136 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.330236 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.330243 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.330246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.330439 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.330911 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.330915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.330918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.331272 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:08.331457 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:08.331955 (+ 498us) server_negotiation.cc:299] Negotiation successful
0924 16:09:08.332171 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260924 16:09:08.332551 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.315975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:41989 (local address 127.0.0.1:60146)
0924 16:09:08.316294 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.316326 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:08.316514 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:08.316957 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:08.316964 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:08.317500 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:08.317765 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:08.317779 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.319102 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.319110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:08.320610 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:08.320629 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.321672 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.321680 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:08.321894 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:08.322718 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:08.322745 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:08.325502 (+ 2757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:08.329373 (+ 3871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:08.329381 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:08.329385 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:08.329996 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:08.330571 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:08.330576 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:08.330580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:08.330760 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:08.331447 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:08.331453 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:08.331666 (+ 213us) client_negotiation.cc:769] Sending connection context
0924 16:09:08.331833 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 16:09:08.332195 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260924 16:09:08.334278 17658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:09:08.335719 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.335164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:43419 (local address 127.0.0.1:47052)
0924 16:09:08.335417 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.335528 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260924 16:09:08.335994 17733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } attempt: 1
I20260924 16:09:08.337016 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.336576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:43419 (local address 127.0.0.1:47066)
0924 16:09:08.336759 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.336859 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260924 16:09:08.337227 17658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:43419: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
I20260924 16:09:08.337392 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.336655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:43419 (local address 127.0.0.1:47082)
0924 16:09:08.336897 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.337043 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 16:09:08.338583 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.338198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:43419 (local address 127.0.0.1:47092)
0924 16:09:08.338364 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.338457 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 16:09:08.338806 17733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:43419: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
I20260924 16:09:08.388511 17658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } attempt: 1
I20260924 16:09:08.390166 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.389476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:43419 (local address 127.0.0.1:47100)
0924 16:09:08.389749 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.389888 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260924 16:09:08.391352 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.390973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:43419 (local address 127.0.0.1:47102)
0924 16:09:08.391135 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.391235 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 16:09:08.391655 17658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:43419: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
I20260924 16:09:08.407070 17733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } attempt: 2
I20260924 16:09:08.408706 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.408060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:43419 (local address 127.0.0.1:47108)
0924 16:09:08.408338 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.408472 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260924 16:09:08.409819 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.409472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:43419 (local address 127.0.0.1:47124)
0924 16:09:08.409621 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.409700 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 16:09:08.410051 17733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:43419: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
I20260924 16:09:08.459970 17658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } attempt: 2
I20260924 16:09:08.461601 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.460884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:43419 (local address 127.0.0.1:47132)
0924 16:09:08.461209 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.461360 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260924 16:09:08.463101 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.462663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:43419 (local address 127.0.0.1:47148)
0924 16:09:08.462877 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.462972 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 16:09:08.463327 17658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:43419: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
I20260924 16:09:08.501410 17733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } attempt: 3
I20260924 16:09:08.503082 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.502408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:43419 (local address 127.0.0.1:47150)
0924 16:09:08.502715 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.502846 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 16:09:08.504305 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.503837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:43419 (local address 127.0.0.1:47154)
0924 16:09:08.504093 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.504188 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
W20260924 16:09:08.504500 17733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:43419: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
I20260924 16:09:08.569640 17658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } attempt: 3
I20260924 16:09:08.571275 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.570672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:43419 (local address 127.0.0.1:47164)
0924 16:09:08.570935 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.571053 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 16:09:08.572690 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.572230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:43419 (local address 127.0.0.1:47178)
0924 16:09:08.572473 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.572563 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260924 16:09:08.572948 17658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:43419: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
I20260924 16:09:08.647753 17733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } attempt: 4
I20260924 16:09:08.649158 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.648611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:43419 (local address 127.0.0.1:47184)
0924 16:09:08.648836 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.648941 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 16:09:08.650687 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.650262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:43419 (local address 127.0.0.1:47188)
0924 16:09:08.650481 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.650569 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260924 16:09:08.650918 17733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:43419: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
W20260924 16:09:08.676776 17741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:08.677150 17741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:08.677297 17741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:08.685094 17741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:09:08.685247 17741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:08.685329 17741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:09:08.685408 17741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:09:08.685469 17741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:09:08.696300 17741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:42019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:35729,127.3.214.253:41989,127.3.214.252:43419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.252.keytab
--principal=kudu/127.3.214.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.252:43419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.214.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:08.697837 17741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:08.699431 17741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:08.714363 17746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:08.714284 17747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:08.714314 17749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:08.715489 17741 server_base.cc:1034] running on GCE node
I20260924 16:09:08.716425 17741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:08.718130 17741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:09:08.719391 17741 hybrid_clock.cc:648] HybridClock initialized: now 1790266148719329 us; error 51 us; skew 500 ppm
I20260924 16:09:08.719810 17741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:08 dist-test-slave-1gc3 krb5kdc[17578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266148, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:08.723399 17741 init.cc:377] Logged in from keytab as kudu/127.3.214.252@KRBTEST.COM (short username kudu)
I20260924 16:09:08.725244 17741 webserver.cc:460] Webserver started at http://127.3.214.252:34773/ using document root <none> and password file <none>
I20260924 16:09:08.726013 17741 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:08.726189 17741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:09:08.726495 17741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:09:08.729846 17741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-2/data/instance:
uuid: "e7431259a39e4fce9d9581268aa584e7"
format_stamp: "Formatted at 2026-09-24 16:09:08 on dist-test-slave-1gc3"
server_key: "2519afa014ad5890e402024e6a5b53d4"
server_key_iv: "f5d0cc8c696a0231fa9fd46e16508897"
server_key_version: "encryptionkey@0"
I20260924 16:09:08.730784 17741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-2/wal/instance:
uuid: "e7431259a39e4fce9d9581268aa584e7"
format_stamp: "Formatted at 2026-09-24 16:09:08 on dist-test-slave-1gc3"
server_key: "2519afa014ad5890e402024e6a5b53d4"
server_key_iv: "f5d0cc8c696a0231fa9fd46e16508897"
server_key_version: "encryptionkey@0"
I20260924 16:09:08.736876 17741 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 16:09:08.741146 17756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:08.742951 17741 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:09:08.743161 17741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-2/wal
uuid: "e7431259a39e4fce9d9581268aa584e7"
format_stamp: "Formatted at 2026-09-24 16:09:08 on dist-test-slave-1gc3"
server_key: "2519afa014ad5890e402024e6a5b53d4"
server_key_iv: "f5d0cc8c696a0231fa9fd46e16508897"
server_key_version: "encryptionkey@0"
I20260924 16:09:08.743393 17741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:08.750198 17658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } attempt: 4
I20260924 16:09:08.751433 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.750943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:43419 (local address 127.0.0.1:47202)
0924 16:09:08.751157 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.751252 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 16:09:08.752672 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.752264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:43419 (local address 127.0.0.1:47218)
0924 16:09:08.752457 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.752545 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260924 16:09:08.752902 17658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:43419: Network error: Client connection negotiation failed: client connection to 127.3.214.252:43419: connect: Connection refused (error 111)
I20260924 16:09:08.761765 17741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:08.766438 17741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:08.766726 17741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:08.786983 17741 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.252:43419
I20260924 16:09:08.787107 17808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.252:43419 every 8 connection(s)
I20260924 16:09:08.788488 17741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-2/data/info.pb
I20260924 16:09:08.790994 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 17741
I20260924 16:09:08.791229 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790265920616583-3931-0/minicluster-data/master-2/wal/instance
I20260924 16:09:08.791707 3931 external_mini_cluster.cc:1428] Setting key 0f33858a3e8772bace282864407179fe
I20260924 16:09:08.794266 17809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:09:08.806252 17809 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35729 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:09:08 dist-test-slave-1gc3 krb5kdc[17578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.252@KRBTEST.COM
Sep 24 16:09:08 dist-test-slave-1gc3 krb5kdc[17578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266148, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.252@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:09:08.822826 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.793922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.3.214.252:43419)
0924 16:09:08.795086 (+ 1164us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:08.795106 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:08.795179 (+ 73us) server_negotiation.cc:407] Connection header received
0924 16:09:08.797916 (+ 2737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:08.797957 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:08.798705 (+ 748us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:08.799348 (+ 643us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:08.800883 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.803388 (+ 2505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.805058 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.806339 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.806595 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:08.812102 (+ 5507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:08.812114 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:08.812150 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:08.812222 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:08.816562 (+ 4340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.817648 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.817658 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.817668 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.817930 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.818448 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.818453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.818456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.819395 (+ 939us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:08.819784 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:08.820426 (+ 642us) server_negotiation.cc:299] Negotiation successful
0924 16:09:08.821103 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":464,"threads_started":1}
I20260924 16:09:08.830142 17813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.809908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39222 (local address 127.3.214.254:35729)
0924 16:09:08.812047 (+ 2139us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:08.812054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:08.812074 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:09:08.812145 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:08.812155 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:08.812248 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:08.812387 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:08.814390 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.815195 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.817329 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.817878 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.817984 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:08.822646 (+ 4662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:08.822652 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:08.822657 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:08.822706 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:08.826576 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.827614 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.827621 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.827624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.827844 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.828308 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.828314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.828317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.828692 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:08.828930 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:08.829414 (+ 484us) server_negotiation.cc:299] Negotiation successful
0924 16:09:08.829666 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1983,"thread_start_us":1764,"threads_started":1}
I20260924 16:09:08.830152 17814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.809877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:35729 (local address 127.0.0.1:39222)
0924 16:09:08.810948 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.811000 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:08.811322 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:08.812562 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:08.812571 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:08.813658 (+ 1087us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:08.814186 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:08.814202 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.815394 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.815400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:08.817140 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:08.817154 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.818136 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.818144 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:08.818303 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:08.819093 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:08.819117 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:08.822348 (+ 3231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:08.826764 (+ 4416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:08.826772 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:08.826778 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:08.827439 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:08.827983 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:08.827988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:08.827991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:08.828134 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:08.828844 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:08.828853 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:08.829086 (+ 233us) client_negotiation.cc:769] Sending connection context
0924 16:09:08.829258 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 16:09:08.829676 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"thread_start_us":575,"threads_started":1}
I20260924 16:09:08.832103 17809 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:09:08 dist-test-slave-1gc3 krb5kdc[17578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
Sep 24 16:09:08 dist-test-slave-1gc3 krb5kdc[17578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266148, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.252@KRBTEST.COM for kudu/127.3.214.253@KRBTEST.COM
I20260924 16:09:08.847323 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.827709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39226 (local address 127.3.214.254:35729)
0924 16:09:08.828517 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:08.828522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:08.828543 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:09:08.828611 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:08.828615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:08.828684 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:08.828797 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:08.830772 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.831857 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.833732 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.834489 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.834641 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:08.840083 (+ 5442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:08.840090 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:08.840094 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:08.840143 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:08.843726 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.844888 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.844894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.844898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.845095 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.845486 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.845492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.845495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.845947 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:08.846310 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:08.846689 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 16:09:08.846931 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":435,"threads_started":1}
I20260924 16:09:08.849867 17814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.833096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:41989 (local address 127.0.0.1:60154)
0924 16:09:08.833506 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.833537 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:08.833689 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:08.835435 (+ 1746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:08.835442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:08.835872 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:08.836044 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:08.836053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.837105 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.837110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:08.838014 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:08.838030 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.839028 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.839034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:08.839279 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:08.839911 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:08.839934 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:08.842458 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:08.846453 (+ 3995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:08.846459 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:08.846463 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:08.847103 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:08.847747 (+ 644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:08.847753 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:08.847757 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:08.848005 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:08.848755 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:08.848759 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:08.848991 (+ 232us) client_negotiation.cc:769] Sending connection context
0924 16:09:08.849166 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 16:09:08.849552 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260924 16:09:08.849932 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.833191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60154 (local address 127.3.214.253:41989)
0924 16:09:08.835070 (+ 1879us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:08.835076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:08.835096 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:09:08.835165 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:08.835169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:08.835251 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:08.835355 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:08.836211 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.836960 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.838272 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.838828 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.838965 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:08.842721 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:08.842729 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:08.842734 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:08.842784 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:08.846264 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.847301 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.847308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.847312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.847565 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.848144 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.848150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.848153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.848570 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:08.848785 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:08.849256 (+ 471us) server_negotiation.cc:299] Negotiation successful
0924 16:09:08.849476 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1734,"thread_start_us":381,"threads_started":1}
I20260924 16:09:08.851225 17809 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:09:08 dist-test-slave-1gc3 krb5kdc[17578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266148, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.252@KRBTEST.COM for kudu/127.3.214.252@KRBTEST.COM
Sep 24 16:09:08 dist-test-slave-1gc3 krb5kdc[17578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.253@KRBTEST.COM
I20260924 16:09:08.865571 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.852153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47238 (local address 127.3.214.252:43419)
0924 16:09:08.852303 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:08.852309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:08.852334 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:09:08.852374 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:08.852376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:08.852444 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:08.852554 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:08.853328 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.854154 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.855406 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.855865 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.856021 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:08.859995 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:08.860000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:08.860004 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:08.860036 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:08.862907 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.863538 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.863543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.863546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.863665 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.864008 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.864013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.864016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.864326 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:08.864464 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:08.864791 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 16:09:08.865093 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":53}
I20260924 16:09:08.865624 17814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.851937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:43419 (local address 127.0.0.1:47238)
0924 16:09:08.852094 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.852114 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:08.852214 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:08.852614 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:08.852618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:08.853003 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:08.853173 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:08.853182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.854292 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.854296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:08.855283 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:08.855299 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.856111 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.856119 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:08.856345 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:08.856937 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:08.856960 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:08.859804 (+ 2844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:08.863037 (+ 3233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:08.863044 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:08.863049 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:08.863433 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:08.863766 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:08.863771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:08.863775 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:08.863913 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:08.864436 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:08.864441 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:08.864585 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 16:09:08.864702 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 16:09:08.865258 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":172}
I20260924 16:09:08.867156 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.850026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60158 (local address 127.3.214.253:41989)
0924 16:09:08.850371 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:08.850377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:08.850680 (+ 303us) server_negotiation.cc:407] Connection header received
0924 16:09:08.850839 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:08.850845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:08.850914 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:08.851006 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:08.852467 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.853323 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.854776 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.855252 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.855365 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:08.860566 (+ 5201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:08.860572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:08.860576 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:08.860618 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:08.863348 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.864443 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.864449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.864451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.864621 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.865048 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.865052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.865054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.865530 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:08.866407 (+ 877us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:08.866578 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 16:09:08.866823 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":879,"threads_started":1}
I20260924 16:09:08.871542 17809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7: Bootstrap starting.
I20260924 16:09:08.872454 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:09:08.875010 17809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7: Neither blocks nor log segments found. Creating new log.
I20260924 16:09:08.876336 17809 log.cc:824] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7: Log is configured to *not* fsync() on all Append() calls
I20260924 16:09:08.880934 17809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7: No bootstrap required, opened a new log
I20260924 16:09:08.893687 17809 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35729 } } peers { permanent_uuid: "43b5a38ca2004e1ca8fd5291880e53de" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } } peers { permanent_uuid: "e7431259a39e4fce9d9581268aa584e7" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } }
I20260924 16:09:08.894788 17809 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:09:08.895037 17809 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7431259a39e4fce9d9581268aa584e7, State: Initialized, Role: FOLLOWER
I20260924 16:09:08.896277 17809 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35729 } } peers { permanent_uuid: "43b5a38ca2004e1ca8fd5291880e53de" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } } peers { permanent_uuid: "e7431259a39e4fce9d9581268aa584e7" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } }
I20260924 16:09:08.898449 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.878264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.3.214.254:35729)
0924 16:09:08.878518 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:08.878524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:08.879949 (+ 1425us) server_negotiation.cc:407] Connection header received
0924 16:09:08.880105 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:08.880111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:08.880206 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:08.880426 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:08.883089 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.884123 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.886378 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.887000 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.887135 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:08.891151 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:08.891158 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:08.891163 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:08.891213 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:08.894935 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.895862 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.895868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.895871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.896026 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.896525 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.896530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.896533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.896916 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:08.897201 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:08.897775 (+ 574us) server_negotiation.cc:299] Negotiation successful
0924 16:09:08.898036 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 16:09:08.899899 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.880028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47240 (local address 127.3.214.252:43419)
0924 16:09:08.880541 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:08.880547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:08.880852 (+ 305us) server_negotiation.cc:407] Connection header received
0924 16:09:08.880939 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:08.880944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:08.881037 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:08.881140 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:08.883690 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.884879 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.887470 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.888306 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.888408 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:08.892359 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:08.892365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:08.892371 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:08.892419 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:08.896270 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.897160 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.897166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.897169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.897366 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.897864 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.897869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.897872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.898359 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:08.898630 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:08.899172 (+ 542us) server_negotiation.cc:299] Negotiation successful
0924 16:09:08.899413 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334}
I20260924 16:09:08.901813 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.878774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60166 (local address 127.3.214.253:41989)
0924 16:09:08.880714 (+ 1940us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:08.880720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:08.880741 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:09:08.880816 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:08.880822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:08.880927 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:08.883416 (+ 2489us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:08.883619 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.884733 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.887365 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.888052 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.892143 (+ 4091us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:08.892480 (+ 337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:08.892486 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:08.892491 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:08.892526 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:08.896334 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.897441 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.897448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.897452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.897634 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.898172 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.898178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.898182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.899199 (+ 1017us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:08.900746 (+ 1547us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:08.900915 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 16:09:08.901389 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1802}
I20260924 16:09:08.902155 17826 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35729 } } peers { permanent_uuid: "43b5a38ca2004e1ca8fd5291880e53de" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } } peers { permanent_uuid: "e7431259a39e4fce9d9581268aa584e7" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } } }
I20260924 16:09:08.902751 17826 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:09:08.904139 17809 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 16:09:08.917363 17837 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:09:08.917528 17837 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:09:08.913818 17741 master_runner.cc:186] Error getting master registration for 127.3.214.254:35729: OK, instance_id { permanent_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357" instance_seqno: 1790266147589002 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:09:08.925840 17741 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:09:08.950206 17733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } attempt: 5
Sep 24 16:09:08 dist-test-slave-1gc3 krb5kdc[17578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266148, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.253@KRBTEST.COM for kudu/127.3.214.252@KRBTEST.COM
I20260924 16:09:08.967669 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.951057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47252 (local address 127.3.214.252:43419)
0924 16:09:08.951252 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:08.951257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:08.951279 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:09:08.951534 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:08.951539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:08.951646 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:08.951749 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:08.952939 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.953834 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.955767 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.956361 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.956504 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:08.961424 (+ 4920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:08.961430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:08.961434 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:08.961481 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:08.964442 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.965455 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.965459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.965462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.965617 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:08.966066 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:08.966069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:08.966071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:08.966451 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:08.966572 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:08.967133 (+ 561us) server_negotiation.cc:299] Negotiation successful
0924 16:09:08.967377 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:09:08.967800 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:08.950865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:43419 (local address 127.0.0.1:47252)
0924 16:09:08.951143 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:08.951173 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:08.951333 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:08.951824 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:08.951829 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:08.952465 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:08.952731 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:08.952745 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.953983 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.953990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:08.955606 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:08.955624 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:08.956546 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:08.956553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:08.956814 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:08.958008 (+ 1194us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:08.958037 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:08.961175 (+ 3138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:08.964627 (+ 3452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:08.964634 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:08.964638 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:08.965300 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:08.965742 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:08.965748 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:08.965752 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:08.965941 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:08.966574 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:08.966579 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:08.966789 (+ 210us) client_negotiation.cc:769] Sending connection context
0924 16:09:08.966977 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 16:09:08.967338 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 16:09:08.974036 17733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de: Bootstrap starting.
I20260924 16:09:08.977847 17733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de: Neither blocks nor log segments found. Creating new log.
I20260924 16:09:08.979175 17733 log.cc:824] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de: Log is configured to *not* fsync() on all Append() calls
I20260924 16:09:08.983253 17733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de: No bootstrap required, opened a new log
I20260924 16:09:08.991779 17733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35729 } } peers { permanent_uuid: "43b5a38ca2004e1ca8fd5291880e53de" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } } peers { permanent_uuid: "e7431259a39e4fce9d9581268aa584e7" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } }
I20260924 16:09:08.992180 17733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:09:08.992275 17733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43b5a38ca2004e1ca8fd5291880e53de, State: Initialized, Role: FOLLOWER
I20260924 16:09:08.992954 17733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35729 } } peers { permanent_uuid: "43b5a38ca2004e1ca8fd5291880e53de" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } } peers { permanent_uuid: "e7431259a39e4fce9d9581268aa584e7" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } }
I20260924 16:09:08.993913 17840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35729 } } peers { permanent_uuid: "43b5a38ca2004e1ca8fd5291880e53de" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } } peers { permanent_uuid: "e7431259a39e4fce9d9581268aa584e7" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } } }
I20260924 16:09:08.994349 17840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:09:08.995276 17733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de [sys.catalog]: configured and running, proceeding with master startup.
W20260924 16:09:09.008105 17851 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:09:09.008412 17851 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 16:09:09.019383 17762 tablet.cc:2369] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:09:09.005558 17666 master_runner.cc:186] Error getting master registration for 127.3.214.254:35729: OK, instance_id { permanent_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357" instance_seqno: 1790266147589002 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:09:09.021184 17666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:09:09.030172 17658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } attempt: 5
Sep 24 16:09:09 dist-test-slave-1gc3 krb5kdc[17578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266147, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for kudu/127.3.214.252@KRBTEST.COM
I20260924 16:09:09.051103 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:09.031135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:43419 (local address 127.0.0.1:47268)
0924 16:09:09.031408 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:09.031434 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:09.031598 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:09.032047 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:09.032054 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:09.032600 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:09.032873 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:09.032886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:09.034496 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:09.034504 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:09.036090 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:09.036107 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:09.038349 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:09.038358 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:09.038607 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:09.039415 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:09.039439 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:09.042668 (+ 3229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:09.047659 (+ 4991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:09.047667 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:09.047672 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:09.048437 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:09.048944 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:09.048950 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:09.048955 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:09.049149 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:09.049877 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:09.049883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:09.050083 (+ 200us) client_negotiation.cc:769] Sending connection context
0924 16:09:09.050275 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 16:09:09.050635 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 16:09:09.052383 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:09.031292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47268 (local address 127.3.214.252:43419)
0924 16:09:09.031493 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:09.031499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:09.031522 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:09:09.031738 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:09.031744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:09.031830 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:09.031951 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:09.033065 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:09.034280 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:09.037359 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:09.038095 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:09.038272 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:09.042986 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:09.042995 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:09.043001 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:09.043058 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:09.047435 (+ 4377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:09.048608 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:09.048623 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:09.048627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:09.048802 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:09.049279 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:09.049285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:09.049288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:09.049731 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:09.049909 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:09.051609 (+ 1700us) server_negotiation.cc:299] Negotiation successful
0924 16:09:09.051939 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 16:09:09.060312 17658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3dd35ee12f44bc93251d1c7d2fe357: Bootstrap starting.
I20260924 16:09:09.065727 17658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f3dd35ee12f44bc93251d1c7d2fe357: Neither blocks nor log segments found. Creating new log.
I20260924 16:09:09.067549 17658 log.cc:824] T 00000000000000000000000000000000 P 3f3dd35ee12f44bc93251d1c7d2fe357: Log is configured to *not* fsync() on all Append() calls
I20260924 16:09:09.072408 17658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3dd35ee12f44bc93251d1c7d2fe357: No bootstrap required, opened a new log
I20260924 16:09:09.086445 17658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f3dd35ee12f44bc93251d1c7d2fe357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35729 } } peers { permanent_uuid: "43b5a38ca2004e1ca8fd5291880e53de" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } } peers { permanent_uuid: "e7431259a39e4fce9d9581268aa584e7" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } }
I20260924 16:09:09.087021 17658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f3dd35ee12f44bc93251d1c7d2fe357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:09:09.087158 17658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f3dd35ee12f44bc93251d1c7d2fe357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3dd35ee12f44bc93251d1c7d2fe357, State: Initialized, Role: FOLLOWER
I20260924 16:09:09.088209 17658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f3dd35ee12f44bc93251d1c7d2fe357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35729 } } peers { permanent_uuid: "43b5a38ca2004e1ca8fd5291880e53de" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } } peers { permanent_uuid: "e7431259a39e4fce9d9581268aa584e7" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } }
I20260924 16:09:09.089063 17826 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:09:09.089462 17826 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35729 } } peers { permanent_uuid: "43b5a38ca2004e1ca8fd5291880e53de" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } } peers { permanent_uuid: "e7431259a39e4fce9d9581268aa584e7" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } }
I20260924 16:09:09.091332 17826 leader_election.cc:288] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f3dd35ee12f44bc93251d1c7d2fe357 (127.3.214.254:35729), 43b5a38ca2004e1ca8fd5291880e53de (127.3.214.253:41989)
I20260924 16:09:09.092804 17854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3dd35ee12f44bc93251d1c7d2fe357 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35729 } } peers { permanent_uuid: "43b5a38ca2004e1ca8fd5291880e53de" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } } peers { permanent_uuid: "e7431259a39e4fce9d9581268aa584e7" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } } }
I20260924 16:09:09.092883 17708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7431259a39e4fce9d9581268aa584e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43b5a38ca2004e1ca8fd5291880e53de" is_pre_election: true
I20260924 16:09:09.093453 17854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3dd35ee12f44bc93251d1c7d2fe357 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:09:09.093807 17708 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7431259a39e4fce9d9581268aa584e7 in term 0.
I20260924 16:09:09.095455 17757 leader_election.cc:302] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43b5a38ca2004e1ca8fd5291880e53de, e7431259a39e4fce9d9581268aa584e7; no voters:
I20260924 16:09:09.096109 17826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:09:09.096292 17826 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:09:09.096422 17826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:09:09.099467 17658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f3dd35ee12f44bc93251d1c7d2fe357 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:09:09.100476 17826 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35729 } } peers { permanent_uuid: "43b5a38ca2004e1ca8fd5291880e53de" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } } peers { permanent_uuid: "e7431259a39e4fce9d9581268aa584e7" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } }
I20260924 16:09:09.105170 17633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7431259a39e4fce9d9581268aa584e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357" is_pre_election: true
I20260924 16:09:09.106076 17633 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3f3dd35ee12f44bc93251d1c7d2fe357 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7431259a39e4fce9d9581268aa584e7 in term 0.
I20260924 16:09:09.108671 17633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7431259a39e4fce9d9581268aa584e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357"
I20260924 16:09:09.108937 17633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f3dd35ee12f44bc93251d1c7d2fe357 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:09:09.113494 17633 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3f3dd35ee12f44bc93251d1c7d2fe357 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7431259a39e4fce9d9581268aa584e7 in term 1.
I20260924 16:09:09.114194 17760 leader_election.cc:302] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f3dd35ee12f44bc93251d1c7d2fe357, e7431259a39e4fce9d9581268aa584e7; no voters:
I20260924 16:09:09.116622 17708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7431259a39e4fce9d9581268aa584e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43b5a38ca2004e1ca8fd5291880e53de"
I20260924 16:09:09.116951 17708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:09:09.120400 17826 leader_election.cc:288] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [CANDIDATE]: Term 1 election: Requested vote from peers 3f3dd35ee12f44bc93251d1c7d2fe357 (127.3.214.254:35729), 43b5a38ca2004e1ca8fd5291880e53de (127.3.214.253:41989)
I20260924 16:09:09.120709 17826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:09:09.120891 17826 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [term 1 LEADER]: Becoming Leader. State: Replica: e7431259a39e4fce9d9581268aa584e7, State: Running, Role: LEADER
I20260924 16:09:09.127669 17826 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35729 } } peers { permanent_uuid: "43b5a38ca2004e1ca8fd5291880e53de" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } } peers { permanent_uuid: "e7431259a39e4fce9d9581268aa584e7" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } }
I20260924 16:09:09.126068 17708 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7431259a39e4fce9d9581268aa584e7 in term 1.
I20260924 16:09:09.147166 17856 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7431259a39e4fce9d9581268aa584e7. Latest consensus state: current_term: 1 leader_uuid: "e7431259a39e4fce9d9581268aa584e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35729 } } peers { permanent_uuid: "43b5a38ca2004e1ca8fd5291880e53de" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } } peers { permanent_uuid: "e7431259a39e4fce9d9581268aa584e7" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } } }
I20260924 16:09:09.147717 17856 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:09:09.151618 17858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:09:09.153752 17858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:09:09.175262 17708 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de [term 1 FOLLOWER]: Refusing update from remote peer e7431259a39e4fce9d9581268aa584e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:09:09.176481 17856 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43b5a38ca2004e1ca8fd5291880e53de" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:09:09.184000 17633 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f3dd35ee12f44bc93251d1c7d2fe357 [term 1 FOLLOWER]: Refusing update from remote peer e7431259a39e4fce9d9581268aa584e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:09:09.184937 17856 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 16:09:09.188133 17687 tablet.cc:2369] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:09:09.225317 17854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3dd35ee12f44bc93251d1c7d2fe357 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7431259a39e4fce9d9581268aa584e7. Latest consensus state: current_term: 1 leader_uuid: "e7431259a39e4fce9d9581268aa584e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35729 } } peers { permanent_uuid: "43b5a38ca2004e1ca8fd5291880e53de" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } } peers { permanent_uuid: "e7431259a39e4fce9d9581268aa584e7" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } } }
I20260924 16:09:09.225709 17854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3dd35ee12f44bc93251d1c7d2fe357 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:09:09.231364 17860 mvcc.cc:205] Tried to move back new op lower bound from 7332930147007373312 to 7332930146858373120. Current Snapshot: MvccSnapshot[applied={T|T < 7332930147007373312}]
I20260924 16:09:09.237044 17874 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 16:09:09.243846 17865 mvcc.cc:205] Tried to move back new op lower bound from 7332930147007373312 to 7332930146858373120. Current Snapshot: MvccSnapshot[applied={T|T < 7332930147007373312}]
I20260924 16:09:09.246749 17840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7431259a39e4fce9d9581268aa584e7. Latest consensus state: current_term: 1 leader_uuid: "e7431259a39e4fce9d9581268aa584e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35729 } } peers { permanent_uuid: "43b5a38ca2004e1ca8fd5291880e53de" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } } peers { permanent_uuid: "e7431259a39e4fce9d9581268aa584e7" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } } }
I20260924 16:09:09.247205 17840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:09:09.251206 17858 catalog_manager.cc:1347] Generated new cluster ID: 4ba4c388a1f6473abc2b9021921faf1c
I20260924 16:09:09.251358 17858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:09:09.256770 17854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3dd35ee12f44bc93251d1c7d2fe357 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7431259a39e4fce9d9581268aa584e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35729 } } peers { permanent_uuid: "43b5a38ca2004e1ca8fd5291880e53de" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } } peers { permanent_uuid: "e7431259a39e4fce9d9581268aa584e7" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } } }
I20260924 16:09:09.257114 17854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3dd35ee12f44bc93251d1c7d2fe357 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:09:09.258415 17862 mvcc.cc:205] Tried to move back new op lower bound from 7332930147007373312 to 7332930146858373120. Current Snapshot: MvccSnapshot[applied={T|T < 7332930147007373312}]
I20260924 16:09:09.257982 17856 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7431259a39e4fce9d9581268aa584e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35729 } } peers { permanent_uuid: "43b5a38ca2004e1ca8fd5291880e53de" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } } peers { permanent_uuid: "e7431259a39e4fce9d9581268aa584e7" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } } }
I20260924 16:09:09.264916 17856 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:09:09.270252 17856 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7431259a39e4fce9d9581268aa584e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35729 } } peers { permanent_uuid: "43b5a38ca2004e1ca8fd5291880e53de" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } } peers { permanent_uuid: "e7431259a39e4fce9d9581268aa584e7" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } } }
I20260924 16:09:09.270752 17856 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:09:09.285157 17840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7431259a39e4fce9d9581268aa584e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3dd35ee12f44bc93251d1c7d2fe357" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 35729 } } peers { permanent_uuid: "43b5a38ca2004e1ca8fd5291880e53de" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 41989 } } peers { permanent_uuid: "e7431259a39e4fce9d9581268aa584e7" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 43419 } } }
I20260924 16:09:09.285485 17840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:09:09.312922 17858 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:09:09.316332 17858 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:09:09.333506 17858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7431259a39e4fce9d9581268aa584e7: Generated new TSK 0
I20260924 16:09:09.334915 17858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:09:10.011157 17851 catalog_manager.cc:1259] Loaded cluster ID: 4ba4c388a1f6473abc2b9021921faf1c
I20260924 16:09:10.011297 17851 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de: loading cluster ID for follower catalog manager: success
I20260924 16:09:10.014176 17851 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de: acquiring CA information for follower catalog manager: success
I20260924 16:09:10.016429 17851 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 43b5a38ca2004e1ca8fd5291880e53de: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:09:10.243913 17874 catalog_manager.cc:1259] Loaded cluster ID: 4ba4c388a1f6473abc2b9021921faf1c
I20260924 16:09:10.244053 17874 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3f3dd35ee12f44bc93251d1c7d2fe357: loading cluster ID for follower catalog manager: success
I20260924 16:09:10.247592 17874 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3f3dd35ee12f44bc93251d1c7d2fe357: acquiring CA information for follower catalog manager: success
I20260924 16:09:10.250229 17874 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3f3dd35ee12f44bc93251d1c7d2fe357: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:09:10.433492 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 17591
I20260924 16:09:10.435247 17653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:10.558938 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 17591
I20260924 16:09:10.577585 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 17666
I20260924 16:09:10.578979 17728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:10.696965 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 17666
I20260924 16:09:10.722406 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 17741
I20260924 16:09:10.723758 17804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:10.840512 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 17741
2026-09-24T16:09:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5123 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:09:10 dist-test-slave-1gc3 krb5kdc[17891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:09:10 dist-test-slave-1gc3 krb5kdc[17891](info): set up 2 sockets
Sep 24 16:09:10 dist-test-slave-1gc3 krb5kdc[17891](info): commencing operation
krb5kdc: starting...
W20260924 16:09:11.904063 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:09:12 dist-test-slave-1gc3 krb5kdc[17891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:09:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:09:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:09:12.198411 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:42013
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:44029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:42013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:09:12.603976 17904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:12.604398 17904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:12.604519 17904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:12.612991 17904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:09:12.613181 17904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:12.613269 17904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:09:12.613355 17904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:09:12.613417 17904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:09:12.625224 17904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:44029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:42013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:42013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:12.627197 17904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:12.628937 17904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:12.643078 17912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:12.643236 17910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:12.643126 17909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:12.644953 17904 server_base.cc:1034] running on GCE node
I20260924 16:09:12.645994 17904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:12.648025 17904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:09:12.649283 17904 hybrid_clock.cc:648] HybridClock initialized: now 1790266152649222 us; error 41 us; skew 500 ppm
I20260924 16:09:12.649780 17904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:12 dist-test-slave-1gc3 krb5kdc[17891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266152, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:12.657753 17904 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:09:12.659945 17904 webserver.cc:460] Webserver started at http://127.3.214.254:42357/ using document root <none> and password file <none>
I20260924 16:09:12.660844 17904 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:12.660965 17904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:09:12.661294 17904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:09:12.665115 17904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "79dd87ef9c3b4857a0566090a0b61b4b"
format_stamp: "Formatted at 2026-09-24 16:09:12 on dist-test-slave-1gc3"
server_key: "bd6bd2b3b3c0fb2d1dbfa5953038dcec"
server_key_iv: "5260b5e7726d18342b71a750f4ae0329"
server_key_version: "encryptionkey@0"
I20260924 16:09:12.666031 17904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "79dd87ef9c3b4857a0566090a0b61b4b"
format_stamp: "Formatted at 2026-09-24 16:09:12 on dist-test-slave-1gc3"
server_key: "bd6bd2b3b3c0fb2d1dbfa5953038dcec"
server_key_iv: "5260b5e7726d18342b71a750f4ae0329"
server_key_version: "encryptionkey@0"
I20260924 16:09:12.673062 17904 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260924 16:09:12.677767 17919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:12.679751 17904 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:09:12.680184 17904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "79dd87ef9c3b4857a0566090a0b61b4b"
format_stamp: "Formatted at 2026-09-24 16:09:12 on dist-test-slave-1gc3"
server_key: "bd6bd2b3b3c0fb2d1dbfa5953038dcec"
server_key_iv: "5260b5e7726d18342b71a750f4ae0329"
server_key_version: "encryptionkey@0"
I20260924 16:09:12.680470 17904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:12.702812 17904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:12.707863 17904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:12.708204 17904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:12.731531 17904 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:42013
I20260924 16:09:12.731570 17971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:42013 every 8 connection(s)
I20260924 16:09:12.733435 17904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:09:12.737907 17972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:09:12.740319 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 17904
I20260924 16:09:12.740561 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:09:12.741345 3931 external_mini_cluster.cc:1428] Setting key 9741f89999ead10737958fbf1a12f6c6
I20260924 16:09:12.752228 17972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79dd87ef9c3b4857a0566090a0b61b4b: Bootstrap starting.
I20260924 16:09:12.756120 17972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79dd87ef9c3b4857a0566090a0b61b4b: Neither blocks nor log segments found. Creating new log.
I20260924 16:09:12.757222 17972 log.cc:824] T 00000000000000000000000000000000 P 79dd87ef9c3b4857a0566090a0b61b4b: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:09:12 dist-test-slave-1gc3 krb5kdc[17891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:09:12.761018 17972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79dd87ef9c3b4857a0566090a0b61b4b: No bootstrap required, opened a new log
I20260924 16:09:12.769304 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:12.743942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48016 (local address 127.3.214.254:42013)
0924 16:09:12.745037 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:12.745059 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:12.745124 (+ 65us) server_negotiation.cc:407] Connection header received
0924 16:09:12.747684 (+ 2560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:12.747719 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:12.748348 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:12.748841 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:12.750543 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:12.752536 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:12.754045 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:12.755150 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:12.755393 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:12.760281 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:12.760293 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:12.760318 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:12.760377 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:12.764133 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:12.765014 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:12.765023 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:12.765032 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:12.765238 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:12.765632 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:12.765636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:12.765638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:12.766470 (+ 832us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:12.766848 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:12.767351 (+ 503us) server_negotiation.cc:299] Negotiation successful
0924 16:09:12.767918 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":444,"threads_started":1}
I20260924 16:09:12.770062 17972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79dd87ef9c3b4857a0566090a0b61b4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79dd87ef9c3b4857a0566090a0b61b4b" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42013 } }
I20260924 16:09:12.770584 17972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79dd87ef9c3b4857a0566090a0b61b4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:09:12.770687 17972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79dd87ef9c3b4857a0566090a0b61b4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79dd87ef9c3b4857a0566090a0b61b4b, State: Initialized, Role: FOLLOWER
I20260924 16:09:12.771402 17972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79dd87ef9c3b4857a0566090a0b61b4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79dd87ef9c3b4857a0566090a0b61b4b" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42013 } }
I20260924 16:09:12.771689 17972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79dd87ef9c3b4857a0566090a0b61b4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:09:12.771831 17972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79dd87ef9c3b4857a0566090a0b61b4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:09:12.772017 17972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79dd87ef9c3b4857a0566090a0b61b4b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:09:12.774299 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:09:12.774792 17972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79dd87ef9c3b4857a0566090a0b61b4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79dd87ef9c3b4857a0566090a0b61b4b" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42013 } }
I20260924 16:09:12.775363 17972 leader_election.cc:302] T 00000000000000000000000000000000 P 79dd87ef9c3b4857a0566090a0b61b4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79dd87ef9c3b4857a0566090a0b61b4b; no voters:
I20260924 16:09:12.776039 17972 leader_election.cc:288] T 00000000000000000000000000000000 P 79dd87ef9c3b4857a0566090a0b61b4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:09:12.776150 17977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79dd87ef9c3b4857a0566090a0b61b4b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:09:12.778805 17972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79dd87ef9c3b4857a0566090a0b61b4b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:09:12.783388 17977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79dd87ef9c3b4857a0566090a0b61b4b [term 1 LEADER]: Becoming Leader. State: Replica: 79dd87ef9c3b4857a0566090a0b61b4b, State: Running, Role: LEADER
I20260924 16:09:12.784441 17977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79dd87ef9c3b4857a0566090a0b61b4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79dd87ef9c3b4857a0566090a0b61b4b" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42013 } }
I20260924 16:09:12.787510 17978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79dd87ef9c3b4857a0566090a0b61b4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79dd87ef9c3b4857a0566090a0b61b4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79dd87ef9c3b4857a0566090a0b61b4b" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42013 } } }
I20260924 16:09:12.787932 17978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79dd87ef9c3b4857a0566090a0b61b4b [sys.catalog]: This master's current role is: LEADER
I20260924 16:09:12.788967 17991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:09:12.788851 17979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79dd87ef9c3b4857a0566090a0b61b4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79dd87ef9c3b4857a0566090a0b61b4b. Latest consensus state: current_term: 1 leader_uuid: "79dd87ef9c3b4857a0566090a0b61b4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79dd87ef9c3b4857a0566090a0b61b4b" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42013 } } }
I20260924 16:09:12.793257 17979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79dd87ef9c3b4857a0566090a0b61b4b [sys.catalog]: This master's current role is: LEADER
I20260924 16:09:12.796761 17991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:09:12.806262 17991 catalog_manager.cc:1347] Generated new cluster ID: 78ee2fc9bcb14ee7bf08e493ab921d8a
I20260924 16:09:12.806408 17991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:09:12.816465 17991 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:09:12.818053 17991 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:09:12.827553 17991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79dd87ef9c3b4857a0566090a0b61b4b: Generated new TSK 0
I20260924 16:09:12.828553 17991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:09:12.889492 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 17904
I20260924 16:09:12.891131 17967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:12.991134 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 17904
2026-09-24T16:09:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2143 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:09:13 dist-test-slave-1gc3 krb5kdc[18000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:09:13 dist-test-slave-1gc3 krb5kdc[18000](info): set up 2 sockets
Sep 24 16:09:13 dist-test-slave-1gc3 krb5kdc[18000](info): commencing operation
krb5kdc: starting...
W20260924 16:09:14.028273 3931 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.005s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:09:14 dist-test-slave-1gc3 krb5kdc[18000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:09:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:09:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:09:14.326361 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:38473
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:38473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:09:14.744112 18013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:14.744551 18013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:14.744707 18013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:14.753124 18013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:09:14.753311 18013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:14.753412 18013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:09:14.753499 18013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:09:14.753561 18013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:09:14.766996 18013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:38473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:38473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:14.768958 18013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:14.770812 18013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:14.786002 18018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:14.786057 18021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:14.786068 18019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:14.787209 18013 server_base.cc:1034] running on GCE node
I20260924 16:09:14.788187 18013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:14.789906 18013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:09:14.791124 18013 hybrid_clock.cc:648] HybridClock initialized: now 1790266154791045 us; error 66 us; skew 500 ppm
I20260924 16:09:14.791522 18013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:14 dist-test-slave-1gc3 krb5kdc[18000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266154, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:14.794831 18013 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:09:14.796597 18013 webserver.cc:460] Webserver started at http://127.3.214.254:44757/ using document root <none> and password file <none>
I20260924 16:09:14.797363 18013 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:14.797472 18013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:09:14.797751 18013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:09:14.801090 18013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "d4533be9c9df4f0584500746c806dcef"
format_stamp: "Formatted at 2026-09-24 16:09:14 on dist-test-slave-1gc3"
server_key: "1ce6ed245d758f00a43ccca33cae1e08"
server_key_iv: "2b5ba859e725d70a2e1a764552ad3737"
server_key_version: "encryptionkey@0"
I20260924 16:09:14.801913 18013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "d4533be9c9df4f0584500746c806dcef"
format_stamp: "Formatted at 2026-09-24 16:09:14 on dist-test-slave-1gc3"
server_key: "1ce6ed245d758f00a43ccca33cae1e08"
server_key_iv: "2b5ba859e725d70a2e1a764552ad3737"
server_key_version: "encryptionkey@0"
I20260924 16:09:14.808189 18013 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260924 16:09:14.812693 18028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:14.814987 18013 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:09:14.815246 18013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "d4533be9c9df4f0584500746c806dcef"
format_stamp: "Formatted at 2026-09-24 16:09:14 on dist-test-slave-1gc3"
server_key: "1ce6ed245d758f00a43ccca33cae1e08"
server_key_iv: "2b5ba859e725d70a2e1a764552ad3737"
server_key_version: "encryptionkey@0"
I20260924 16:09:14.815495 18013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:14.840642 18013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:14.845554 18013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:14.845849 18013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:14.869234 18013 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:38473
I20260924 16:09:14.869324 18079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:38473 every 8 connection(s)
I20260924 16:09:14.870904 18013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:09:14.876372 18080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:09:14.881031 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 18013
I20260924 16:09:14.881335 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:09:14.882344 3931 external_mini_cluster.cc:1428] Setting key 36ccc70e775fa52a8e16e68916843422
I20260924 16:09:14.893298 18080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef: Bootstrap starting.
I20260924 16:09:14.897635 18080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef: Neither blocks nor log segments found. Creating new log.
Sep 24 16:09:14 dist-test-slave-1gc3 krb5kdc[18000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:09:14.899118 18080 log.cc:824] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef: Log is configured to *not* fsync() on all Append() calls
I20260924 16:09:14.903626 18080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef: No bootstrap required, opened a new log
I20260924 16:09:14.912359 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:14.885368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44352 (local address 127.3.214.254:38473)
0924 16:09:14.886572 (+ 1204us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:14.886591 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:14.886645 (+ 54us) server_negotiation.cc:407] Connection header received
0924 16:09:14.888660 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:14.888691 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:14.889233 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:14.889673 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:14.891308 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:14.892758 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:14.894230 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:14.894999 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:14.895212 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:14.900446 (+ 5234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:14.900462 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:14.900491 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:14.900544 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:14.904487 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:14.905953 (+ 1466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:14.905963 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:14.905976 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:14.906373 (+ 397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:14.906973 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:14.906977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:14.906979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:14.908717 (+ 1738us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:14.909086 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:14.909854 (+ 768us) server_negotiation.cc:299] Negotiation successful
0924 16:09:14.910742 (+ 888us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":498,"threads_started":1}
I20260924 16:09:14.914603 18080 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4533be9c9df4f0584500746c806dcef" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38473 } }
I20260924 16:09:14.915158 18080 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:09:14.915274 18080 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4533be9c9df4f0584500746c806dcef, State: Initialized, Role: FOLLOWER
I20260924 16:09:14.916034 18080 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4533be9c9df4f0584500746c806dcef" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38473 } }
I20260924 16:09:14.916318 18080 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:09:14.916462 18080 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:09:14.916644 18080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:09:14.919682 18080 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4533be9c9df4f0584500746c806dcef" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38473 } }
I20260924 16:09:14.920425 18080 leader_election.cc:302] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4533be9c9df4f0584500746c806dcef; no voters:
I20260924 16:09:14.921269 18080 leader_election.cc:288] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:09:14.921466 18085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:09:14.922335 18085 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 1 LEADER]: Becoming Leader. State: Replica: d4533be9c9df4f0584500746c806dcef, State: Running, Role: LEADER
I20260924 16:09:14.923032 18085 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4533be9c9df4f0584500746c806dcef" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38473 } }
I20260924 16:09:14.923992 18080 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:09:14.926019 18087 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4533be9c9df4f0584500746c806dcef. Latest consensus state: current_term: 1 leader_uuid: "d4533be9c9df4f0584500746c806dcef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4533be9c9df4f0584500746c806dcef" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38473 } } }
I20260924 16:09:14.926812 18087 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [sys.catalog]: This master's current role is: LEADER
I20260924 16:09:14.926314 18086 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4533be9c9df4f0584500746c806dcef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4533be9c9df4f0584500746c806dcef" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38473 } } }
I20260924 16:09:14.927156 18086 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [sys.catalog]: This master's current role is: LEADER
I20260924 16:09:14.929685 18090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:09:14.935832 18090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:09:14.946475 18090 catalog_manager.cc:1347] Generated new cluster ID: c59b21c5970c42ad80cc9fb7e03a9006
I20260924 16:09:14.946601 18090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:09:14.959631 18090 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:09:14.961177 18090 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:09:14.970902 18090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef: Generated new TSK 0
I20260924 16:09:14.971899 18090 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.214.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
I20260924 16:09:15.052390 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:38473
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:09:15.440861 18106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:15.441203 18106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:15.441332 18106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:15.449010 18106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:15.449172 18106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:09:15.449323 18106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:09:15.459661 18106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:38473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:15.461067 18106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:15.462538 18106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:15.478087 18111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:15.478166 18112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:15.478188 18114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:15.479151 18106 server_base.cc:1034] running on GCE node
I20260924 16:09:15.479982 18106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:15.481233 18106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:09:15.482448 18106 hybrid_clock.cc:648] HybridClock initialized: now 1790266155482371 us; error 70 us; skew 500 ppm
I20260924 16:09:15.482821 18106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:15 dist-test-slave-1gc3 krb5kdc[18000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266155, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:15.486667 18106 init.cc:377] Logged in from keytab as kudu/127.3.214.193@KRBTEST.COM (short username kudu)
I20260924 16:09:15.488577 18106 webserver.cc:460] Webserver started at http://127.3.214.193:37543/ using document root <none> and password file <none>
I20260924 16:09:15.489500 18106 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:15.489638 18106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:09:15.489987 18106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:09:15.493625 18106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "f1652cd36cab4e51a07e4d1fb6e361df"
format_stamp: "Formatted at 2026-09-24 16:09:15 on dist-test-slave-1gc3"
server_key: "ef75e92240e9f99c8fef941bf343a39b"
server_key_iv: "07ad02358a7722f95df2ca8b7b1309c2"
server_key_version: "encryptionkey@0"
I20260924 16:09:15.494565 18106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "f1652cd36cab4e51a07e4d1fb6e361df"
format_stamp: "Formatted at 2026-09-24 16:09:15 on dist-test-slave-1gc3"
server_key: "ef75e92240e9f99c8fef941bf343a39b"
server_key_iv: "07ad02358a7722f95df2ca8b7b1309c2"
server_key_version: "encryptionkey@0"
I20260924 16:09:15.501513 18106 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 16:09:15.506357 18121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:15.508183 18106 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260924 16:09:15.508443 18106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "f1652cd36cab4e51a07e4d1fb6e361df"
format_stamp: "Formatted at 2026-09-24 16:09:15 on dist-test-slave-1gc3"
server_key: "ef75e92240e9f99c8fef941bf343a39b"
server_key_iv: "07ad02358a7722f95df2ca8b7b1309c2"
server_key_version: "encryptionkey@0"
I20260924 16:09:15.508674 18106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:15.533509 18106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:15.539180 18106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:15.539528 18106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:15.540824 18106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:09:15.542640 18106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:09:15.542737 18106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:15.542886 18106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:09:15.542943 18106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:15.579742 18106 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:34359
I20260924 16:09:15.579818 18233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:34359 every 8 connection(s)
I20260924 16:09:15.581481 18106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:09:15.589494 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 18106
I20260924 16:09:15.589762 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:09:15.590547 3931 external_mini_cluster.cc:1428] Setting key c55fc3086ac3d3b6a5c5be31d96989b1
Sep 24 16:09:15 dist-test-slave-1gc3 krb5kdc[18000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266155, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:09:15.606444 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:15.584564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:53433 (local address 127.3.214.254:38473)
0924 16:09:15.585091 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:15.585096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:15.586282 (+ 1186us) server_negotiation.cc:407] Connection header received
0924 16:09:15.588602 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:15.588610 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:15.588713 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:15.588822 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:15.591148 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:15.592110 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:15.594208 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:15.594859 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:15.595015 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:15.598963 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:15.598971 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:15.598976 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:15.599027 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:15.602559 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:15.603479 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:15.603485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:15.603488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:15.603657 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:15.604165 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:15.604169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:15.604172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:15.604589 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:15.604723 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:15.605814 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0924 16:09:15.606067 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":269,"threads_started":1}
I20260924 16:09:15.607772 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:15.585014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.193:53433)
0924 16:09:15.586010 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:15.586088 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:15.588232 (+ 2144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:15.589160 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:15.589170 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:15.589967 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:15.590914 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:15.590941 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:15.592412 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:15.592418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:15.593976 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:15.593998 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:15.595166 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:15.595173 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:15.595447 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:15.596210 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:15.596243 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:15.598612 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:15.602776 (+ 4164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:15.602786 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:15.602802 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:15.603331 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:15.603803 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:15.603807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:15.603810 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:15.604039 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:15.604760 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:15.604769 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:15.605294 (+ 525us) client_negotiation.cc:769] Sending connection context
0924 16:09:15.605657 (+ 363us) client_negotiation.cc:240] Negotiation successful
0924 16:09:15.606327 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":376,"threads_started":1}
I20260924 16:09:15.609740 18234 heartbeater.cc:347] Connected to a master server at 127.3.214.254:38473
I20260924 16:09:15.610252 18234 heartbeater.cc:464] Registering TS with master...
I20260924 16:09:15.611281 18234 heartbeater.cc:511] Master 127.3.214.254:38473 requested a full tablet report, sending...
I20260924 16:09:15.614300 18045 ts_manager.cc:194] Registered new tserver with Master: f1652cd36cab4e51a07e4d1fb6e361df (127.3.214.193:34359)
I20260924 16:09:15.616145 18045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:53433
WARNING: no policy specified for kudu/127.3.214.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
I20260924 16:09:15.655292 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:38473
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:09:16.005470 18240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:16.005792 18240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:16.005931 18240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:16.013501 18240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:16.013648 18240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:09:16.013823 18240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:09:16.024436 18240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:38473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:16.025807 18240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:16.027196 18240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:16.041536 18245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:16.041498 18248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:16.041553 18246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:16.042378 18240 server_base.cc:1034] running on GCE node
I20260924 16:09:16.043172 18240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:16.044456 18240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:09:16.045679 18240 hybrid_clock.cc:648] HybridClock initialized: now 1790266156045596 us; error 68 us; skew 500 ppm
I20260924 16:09:16.046089 18240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:16 dist-test-slave-1gc3 krb5kdc[18000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266156, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:16.050011 18240 init.cc:377] Logged in from keytab as kudu/127.3.214.194@KRBTEST.COM (short username kudu)
I20260924 16:09:16.051815 18240 webserver.cc:460] Webserver started at http://127.3.214.194:35369/ using document root <none> and password file <none>
I20260924 16:09:16.052724 18240 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:16.052845 18240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:09:16.053211 18240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:09:16.057045 18240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "4252c8af4a7745c3b0eef21e54f18827"
format_stamp: "Formatted at 2026-09-24 16:09:16 on dist-test-slave-1gc3"
server_key: "1ec67842ee8569017b0974650e36392f"
server_key_iv: "f0e5a6084269af7b6754a4d0f27ecce6"
server_key_version: "encryptionkey@0"
I20260924 16:09:16.057961 18240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "4252c8af4a7745c3b0eef21e54f18827"
format_stamp: "Formatted at 2026-09-24 16:09:16 on dist-test-slave-1gc3"
server_key: "1ec67842ee8569017b0974650e36392f"
server_key_iv: "f0e5a6084269af7b6754a4d0f27ecce6"
server_key_version: "encryptionkey@0"
I20260924 16:09:16.064677 18240 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 16:09:16.069315 18256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:16.071151 18240 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:09:16.071363 18240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "4252c8af4a7745c3b0eef21e54f18827"
format_stamp: "Formatted at 2026-09-24 16:09:16 on dist-test-slave-1gc3"
server_key: "1ec67842ee8569017b0974650e36392f"
server_key_iv: "f0e5a6084269af7b6754a4d0f27ecce6"
server_key_version: "encryptionkey@0"
I20260924 16:09:16.071600 18240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:16.095857 18240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:16.101385 18240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:16.101804 18240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:16.103096 18240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:09:16.104740 18240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:09:16.104821 18240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:16.104960 18240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:09:16.105026 18240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:16.140746 18240 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:42017
I20260924 16:09:16.140830 18368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:42017 every 8 connection(s)
I20260924 16:09:16.142379 18240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:09:16.146270 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 18240
I20260924 16:09:16.146503 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:09:16.147181 3931 external_mini_cluster.cc:1428] Setting key 34ec5268c4af432b51235e4f241c1305
Sep 24 16:09:16 dist-test-slave-1gc3 krb5kdc[18000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266156, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:09:16.168767 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:16.145329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:58569 (local address 127.3.214.254:38473)
0924 16:09:16.145797 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:16.145802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:16.147494 (+ 1692us) server_negotiation.cc:407] Connection header received
0924 16:09:16.150082 (+ 2588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:16.150086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:16.150212 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:16.150329 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:16.152789 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:16.153609 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:16.155971 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:16.156495 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:16.156633 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:16.161442 (+ 4809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:16.161449 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:16.161453 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:16.161502 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:16.164928 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:16.165858 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:16.165863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:16.165866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:16.166074 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:16.166673 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:16.166678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:16.166681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:16.167037 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:16.167183 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:16.168100 (+ 917us) server_negotiation.cc:299] Negotiation successful
0924 16:09:16.168367 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":249,"threads_started":1}
I20260924 16:09:16.169972 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:16.146004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.194:58569)
0924 16:09:16.147238 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:16.147312 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:16.149729 (+ 2417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:16.150692 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:16.150707 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:16.151537 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:16.152546 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:16.152577 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:16.153785 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:16.153791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:16.155761 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:16.155780 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:16.156797 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:16.156804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:16.157131 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:16.158003 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:16.158041 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:16.161071 (+ 3030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:16.165120 (+ 4049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:16.165130 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:16.165150 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:16.165694 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:16.166267 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:16.166271 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:16.166274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:16.166528 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:16.167208 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:16.167215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:16.167667 (+ 452us) client_negotiation.cc:769] Sending connection context
0924 16:09:16.167965 (+ 298us) client_negotiation.cc:240] Negotiation successful
0924 16:09:16.168496 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":417,"threads_started":1}
I20260924 16:09:16.171900 18369 heartbeater.cc:347] Connected to a master server at 127.3.214.254:38473
I20260924 16:09:16.172390 18369 heartbeater.cc:464] Registering TS with master...
I20260924 16:09:16.173493 18369 heartbeater.cc:511] Master 127.3.214.254:38473 requested a full tablet report, sending...
I20260924 16:09:16.175586 18045 ts_manager.cc:194] Registered new tserver with Master: 4252c8af4a7745c3b0eef21e54f18827 (127.3.214.194:42017)
I20260924 16:09:16.176664 18045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:58569
WARNING: no policy specified for kudu/127.3.214.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
I20260924 16:09:16.214519 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:38473
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:09:16.611979 18375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:16.612387 18375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:16.612515 18375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:16.620606 18375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:16.620850 18375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260924 16:09:16.620963 18234 heartbeater.cc:503] Master 127.3.214.254:38473 was elected leader, sending a full tablet report...
W20260924 16:09:16.621055 18375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:09:16.632758 18375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:38473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:16.634552 18375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:16.636209 18375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:16.652174 18380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:16.652226 18381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:16.652273 18383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:16.653258 18375 server_base.cc:1034] running on GCE node
I20260924 16:09:16.654039 18375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:16.655242 18375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:09:16.656607 18375 hybrid_clock.cc:648] HybridClock initialized: now 1790266156656540 us; error 44 us; skew 500 ppm
I20260924 16:09:16.657019 18375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:16 dist-test-slave-1gc3 krb5kdc[18000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266156, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:16.660674 18375 init.cc:377] Logged in from keytab as kudu/127.3.214.195@KRBTEST.COM (short username kudu)
I20260924 16:09:16.662583 18375 webserver.cc:460] Webserver started at http://127.3.214.195:35891/ using document root <none> and password file <none>
I20260924 16:09:16.663408 18375 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:16.663533 18375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:09:16.663843 18375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:09:16.667038 18375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "065aa5432a694522acaa7a486a720a31"
format_stamp: "Formatted at 2026-09-24 16:09:16 on dist-test-slave-1gc3"
server_key: "13e70b2a461cc85d44c301125c5735b0"
server_key_iv: "3f66d4de4d6fa2817e9cbacab03ae81e"
server_key_version: "encryptionkey@0"
I20260924 16:09:16.667817 18375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "065aa5432a694522acaa7a486a720a31"
format_stamp: "Formatted at 2026-09-24 16:09:16 on dist-test-slave-1gc3"
server_key: "13e70b2a461cc85d44c301125c5735b0"
server_key_iv: "3f66d4de4d6fa2817e9cbacab03ae81e"
server_key_version: "encryptionkey@0"
I20260924 16:09:16.674070 18375 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 16:09:16.678509 18390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:16.680265 18375 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:09:16.680455 18375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "065aa5432a694522acaa7a486a720a31"
format_stamp: "Formatted at 2026-09-24 16:09:16 on dist-test-slave-1gc3"
server_key: "13e70b2a461cc85d44c301125c5735b0"
server_key_iv: "3f66d4de4d6fa2817e9cbacab03ae81e"
server_key_version: "encryptionkey@0"
I20260924 16:09:16.680658 18375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:16.721621 18375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:16.726598 18375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:16.726903 18375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:16.728060 18375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:09:16.729681 18375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:09:16.729785 18375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:16.729910 18375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:09:16.729962 18375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:16.767500 18375 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:42317
I20260924 16:09:16.767624 18502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:42317 every 8 connection(s)
I20260924 16:09:16.768913 18375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:09:16.773877 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 18375
I20260924 16:09:16.774133 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:09:16.774753 3931 external_mini_cluster.cc:1428] Setting key 39cd21006c36e2776ee92b38767d1f9a
Sep 24 16:09:16 dist-test-slave-1gc3 krb5kdc[18000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266156, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:09:16.794175 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:16.772301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:36215 (local address 127.3.214.254:38473)
0924 16:09:16.772860 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:16.772865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:16.774271 (+ 1406us) server_negotiation.cc:407] Connection header received
0924 16:09:16.776981 (+ 2710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:16.776987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:16.777075 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:16.777184 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:16.779471 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:16.780285 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:16.782147 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:16.782571 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:16.782691 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:16.787166 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:16.787173 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:16.787177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:16.787225 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:16.790210 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:16.791367 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:16.791372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:16.791375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:16.791513 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:16.792063 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:16.792069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:16.792071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:16.792484 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:16.792678 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:16.793533 (+ 855us) server_negotiation.cc:299] Negotiation successful
0924 16:09:16.793805 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":257,"threads_started":1}
I20260924 16:09:16.795149 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:16.772927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.195:36215)
0924 16:09:16.773962 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:16.774044 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:16.776478 (+ 2434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:16.777642 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:16.777654 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:16.778488 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:16.779270 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:16.779293 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:16.780474 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:16.780479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:16.781929 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:16.781945 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:16.782829 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:16.782835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:16.783103 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:16.783924 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:16.783954 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:16.786856 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:16.790421 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:16.790435 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:16.790458 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:16.791215 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:16.791627 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:16.791633 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:16.791637 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:16.791885 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:16.792645 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:16.792652 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:16.793043 (+ 391us) client_negotiation.cc:769] Sending connection context
0924 16:09:16.793335 (+ 292us) client_negotiation.cc:240] Negotiation successful
0924 16:09:16.793802 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":350,"threads_started":1}
I20260924 16:09:16.796865 18503 heartbeater.cc:347] Connected to a master server at 127.3.214.254:38473
I20260924 16:09:16.797228 18503 heartbeater.cc:464] Registering TS with master...
I20260924 16:09:16.798056 18503 heartbeater.cc:511] Master 127.3.214.254:38473 requested a full tablet report, sending...
I20260924 16:09:16.799832 18045 ts_manager.cc:194] Registered new tserver with Master: 065aa5432a694522acaa7a486a720a31 (127.3.214.195:42317)
I20260924 16:09:16.800801 18045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:36215
I20260924 16:09:16.804378 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:09:16.825206 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:16.809566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44360 (local address 127.3.214.254:38473)
0924 16:09:16.809772 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:16.809779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:16.810256 (+ 477us) server_negotiation.cc:407] Connection header received
0924 16:09:16.810466 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:16.810472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:16.810555 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:16.810647 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:16.812193 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:16.813262 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:16.814939 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:16.815419 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:16.815558 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:16.818454 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:16.818460 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:16.818464 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:16.818511 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:16.821555 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:16.822433 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:16.822438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:16.822440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:16.822589 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:16.823114 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:16.823121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:16.823123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:16.823704 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:16.824040 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:16.824511 (+ 471us) server_negotiation.cc:299] Negotiation successful
0924 16:09:16.824803 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:09:16.830730 18045 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:09:16.839637 18045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:09:16.871439 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:16.857047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:34359 (local address 127.0.0.1:34912)
0924 16:09:16.857995 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:16.858050 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:16.858305 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:16.862321 (+ 4016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:16.862326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:16.862356 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:09:16.862977 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:16.862994 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:16.864901 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:16.864907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:16.867539 (+ 2632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:16.867558 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:16.869876 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:16.869885 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:16.870334 (+ 449us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:16.870462 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 16:09:16.870620 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 16:09:16.870896 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":177,"thread_start_us":404,"threads_started":1}
I20260924 16:09:16.872129 18521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:16.860552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51558 (local address 127.3.214.194:42017)
0924 16:09:16.861290 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:16.861303 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:16.862225 (+ 922us) server_negotiation.cc:407] Connection header received
0924 16:09:16.862369 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:16.862378 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:16.862734 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:16.862950 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:09:16.863906 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:16.865268 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:16.868002 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:16.869267 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:16.869427 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:16.869659 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:16.871591 (+ 1932us) server_negotiation.cc:299] Negotiation successful
0924 16:09:16.871788 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":409,"threads_started":1}
I20260924 16:09:16.872210 18518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:16.857316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:42317 (local address 127.0.0.1:33296)
0924 16:09:16.859636 (+ 2320us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:16.859664 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:16.859844 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:16.861611 (+ 1767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:16.861622 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:16.861670 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:09:16.862734 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:16.862750 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:16.864472 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:16.864478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:16.867101 (+ 2623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:16.867122 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:16.869926 (+ 2804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:16.869934 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:16.870330 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:16.870455 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 16:09:16.870620 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 16:09:16.871791 (+ 1171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2121,"mutex_wait_us":1024,"thread_start_us":1920,"threads_started":1}
I20260924 16:09:16.873090 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:16.857402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34912 (local address 127.3.214.193:34359)
0924 16:09:16.860476 (+ 3074us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:16.860488 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:16.860525 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:09:16.860668 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:16.860677 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:16.860977 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:16.862007 (+ 1030us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:09:16.863254 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:16.864729 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:16.867762 (+ 3033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:16.869307 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:16.869445 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:16.869663 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:16.872497 (+ 2834us) server_negotiation.cc:299] Negotiation successful
0924 16:09:16.872729 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2883,"thread_start_us":2724,"threads_started":1}
I20260924 16:09:16.873116 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:16.858276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33296 (local address 127.3.214.195:42317)
0924 16:09:16.860972 (+ 2696us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:16.860980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:16.861012 (+ 32us) server_negotiation.cc:407] Connection header received
0924 16:09:16.861120 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:16.861126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:16.861325 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:16.862277 (+ 952us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:09:16.862964 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:16.864309 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:16.867331 (+ 3022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:16.869165 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:16.869316 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:16.869643 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:16.870839 (+ 1196us) server_negotiation.cc:299] Negotiation successful
0924 16:09:16.872707 (+ 1868us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2498,"thread_start_us":2294,"threads_started":1}
I20260924 16:09:16.873718 18520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:16.859832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:42017 (local address 127.0.0.1:51558)
0924 16:09:16.861861 (+ 2029us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:16.861884 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:16.862046 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:16.863190 (+ 1144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:16.863195 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:16.863222 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:09:16.863663 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:16.863674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:16.865503 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:16.865508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:16.867835 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:16.867847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:16.869985 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:16.869994 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:16.871130 (+ 1136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:16.871171 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 16:09:16.871263 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 16:09:16.871439 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1876,"mutex_wait_us":798,"thread_start_us":1670,"threads_started":1}
I20260924 16:09:16.877849 18169 tablet_service.cc:1467] Processing CreateTablet for tablet 6fc1e4f2c9bf43bbada44ca50d14433a (DEFAULT_TABLE table=test-table [id=2c832d329cca4260bcc41949d2ea2c26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:09:16.878306 18438 tablet_service.cc:1467] Processing CreateTablet for tablet 6fc1e4f2c9bf43bbada44ca50d14433a (DEFAULT_TABLE table=test-table [id=2c832d329cca4260bcc41949d2ea2c26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:09:16.878306 18304 tablet_service.cc:1467] Processing CreateTablet for tablet 6fc1e4f2c9bf43bbada44ca50d14433a (DEFAULT_TABLE table=test-table [id=2c832d329cca4260bcc41949d2ea2c26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:09:16.879724 18169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fc1e4f2c9bf43bbada44ca50d14433a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:09:16.880158 18304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fc1e4f2c9bf43bbada44ca50d14433a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:09:16.880155 18438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fc1e4f2c9bf43bbada44ca50d14433a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:09:16.895994 18522 tablet_bootstrap.cc:492] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df: Bootstrap starting.
I20260924 16:09:16.902849 18522 tablet_bootstrap.cc:654] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df: Neither blocks nor log segments found. Creating new log.
I20260924 16:09:16.902913 18524 tablet_bootstrap.cc:492] T 6fc1e4f2c9bf43bbada44ca50d14433a P 4252c8af4a7745c3b0eef21e54f18827: Bootstrap starting.
I20260924 16:09:16.904827 18523 tablet_bootstrap.cc:492] T 6fc1e4f2c9bf43bbada44ca50d14433a P 065aa5432a694522acaa7a486a720a31: Bootstrap starting.
I20260924 16:09:16.905081 18522 log.cc:824] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df: Log is configured to *not* fsync() on all Append() calls
I20260924 16:09:16.907095 18524 tablet_bootstrap.cc:654] T 6fc1e4f2c9bf43bbada44ca50d14433a P 4252c8af4a7745c3b0eef21e54f18827: Neither blocks nor log segments found. Creating new log.
I20260924 16:09:16.908488 18523 tablet_bootstrap.cc:654] T 6fc1e4f2c9bf43bbada44ca50d14433a P 065aa5432a694522acaa7a486a720a31: Neither blocks nor log segments found. Creating new log.
I20260924 16:09:16.908598 18524 log.cc:824] T 6fc1e4f2c9bf43bbada44ca50d14433a P 4252c8af4a7745c3b0eef21e54f18827: Log is configured to *not* fsync() on all Append() calls
I20260924 16:09:16.909761 18523 log.cc:824] T 6fc1e4f2c9bf43bbada44ca50d14433a P 065aa5432a694522acaa7a486a720a31: Log is configured to *not* fsync() on all Append() calls
I20260924 16:09:16.910950 18522 tablet_bootstrap.cc:492] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df: No bootstrap required, opened a new log
I20260924 16:09:16.911295 18522 ts_tablet_manager.cc:1397] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.000s
I20260924 16:09:16.912724 18524 tablet_bootstrap.cc:492] T 6fc1e4f2c9bf43bbada44ca50d14433a P 4252c8af4a7745c3b0eef21e54f18827: No bootstrap required, opened a new log
I20260924 16:09:16.913132 18524 ts_tablet_manager.cc:1397] T 6fc1e4f2c9bf43bbada44ca50d14433a P 4252c8af4a7745c3b0eef21e54f18827: Time spent bootstrapping tablet: real 0.010s user 0.010s sys 0.000s
I20260924 16:09:16.920380 18522 raft_consensus.cc:348] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } } peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } }
I20260924 16:09:16.920959 18522 raft_consensus.cc:374] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:09:16.921087 18522 raft_consensus.cc:729] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1652cd36cab4e51a07e4d1fb6e361df, State: Initialized, Role: FOLLOWER
I20260924 16:09:16.921831 18522 consensus_queue.cc:260] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } } peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } }
I20260924 16:09:16.923122 18523 tablet_bootstrap.cc:492] T 6fc1e4f2c9bf43bbada44ca50d14433a P 065aa5432a694522acaa7a486a720a31: No bootstrap required, opened a new log
I20260924 16:09:16.923491 18523 ts_tablet_manager.cc:1397] T 6fc1e4f2c9bf43bbada44ca50d14433a P 065aa5432a694522acaa7a486a720a31: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.009s
I20260924 16:09:16.924196 18522 ts_tablet_manager.cc:1428] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260924 16:09:16.925630 18524 raft_consensus.cc:348] T 6fc1e4f2c9bf43bbada44ca50d14433a P 4252c8af4a7745c3b0eef21e54f18827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } } peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } }
I20260924 16:09:16.926308 18524 raft_consensus.cc:374] T 6fc1e4f2c9bf43bbada44ca50d14433a P 4252c8af4a7745c3b0eef21e54f18827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:09:16.926427 18524 raft_consensus.cc:729] T 6fc1e4f2c9bf43bbada44ca50d14433a P 4252c8af4a7745c3b0eef21e54f18827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4252c8af4a7745c3b0eef21e54f18827, State: Initialized, Role: FOLLOWER
I20260924 16:09:16.927203 18524 consensus_queue.cc:260] T 6fc1e4f2c9bf43bbada44ca50d14433a P 4252c8af4a7745c3b0eef21e54f18827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } } peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } }
I20260924 16:09:16.928926 18369 heartbeater.cc:503] Master 127.3.214.254:38473 was elected leader, sending a full tablet report...
I20260924 16:09:16.929531 18524 ts_tablet_manager.cc:1428] T 6fc1e4f2c9bf43bbada44ca50d14433a P 4252c8af4a7745c3b0eef21e54f18827: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260924 16:09:16.933094 18523 raft_consensus.cc:348] T 6fc1e4f2c9bf43bbada44ca50d14433a P 065aa5432a694522acaa7a486a720a31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } } peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } }
I20260924 16:09:16.933504 18523 raft_consensus.cc:374] T 6fc1e4f2c9bf43bbada44ca50d14433a P 065aa5432a694522acaa7a486a720a31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:09:16.933651 18523 raft_consensus.cc:729] T 6fc1e4f2c9bf43bbada44ca50d14433a P 065aa5432a694522acaa7a486a720a31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 065aa5432a694522acaa7a486a720a31, State: Initialized, Role: FOLLOWER
I20260924 16:09:16.934307 18523 consensus_queue.cc:260] T 6fc1e4f2c9bf43bbada44ca50d14433a P 065aa5432a694522acaa7a486a720a31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } } peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } }
I20260924 16:09:16.935544 18503 heartbeater.cc:503] Master 127.3.214.254:38473 was elected leader, sending a full tablet report...
I20260924 16:09:16.936159 18523 ts_tablet_manager.cc:1428] T 6fc1e4f2c9bf43bbada44ca50d14433a P 065aa5432a694522acaa7a486a720a31: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
W20260924 16:09:17.021068 18504 tablet.cc:2369] T 6fc1e4f2c9bf43bbada44ca50d14433a P 065aa5432a694522acaa7a486a720a31: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:09:17.084901 18235 tablet.cc:2369] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:09:17.144977 18370 tablet.cc:2369] T 6fc1e4f2c9bf43bbada44ca50d14433a P 4252c8af4a7745c3b0eef21e54f18827: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:09:17.171916 18528 raft_consensus.cc:482] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:09:17.172281 18528 raft_consensus.cc:504] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } } peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } }
I20260924 16:09:17.174152 18528 leader_election.cc:288] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 065aa5432a694522acaa7a486a720a31 (127.3.214.195:42317), 4252c8af4a7745c3b0eef21e54f18827 (127.3.214.194:42017)
I20260924 16:09:17.182188 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:17.174767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:50303 (local address 127.3.214.195:42317)
0924 16:09:17.175009 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:17.175014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:17.175366 (+ 352us) server_negotiation.cc:407] Connection header received
0924 16:09:17.175708 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:17.175717 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:17.175819 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:17.175883 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:09:17.176775 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:17.177825 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:17.180078 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:17.181036 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:17.181191 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:17.181261 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:17.181686 (+ 425us) server_negotiation.cc:299] Negotiation successful
0924 16:09:17.181852 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:09:17.182258 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:17.174551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:42317 (local address 127.3.214.193:50303)
0924 16:09:17.175258 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:17.175278 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:17.175436 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:17.176113 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:17.176117 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:17.176145 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:09:17.176570 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:17.176596 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:17.177976 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:17.177981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:17.179878 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:17.179893 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:17.181280 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:17.181286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:17.181472 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:17.181513 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 16:09:17.181614 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 16:09:17.181742 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":52,"thread_start_us":382,"threads_started":1}
I20260924 16:09:17.182322 18521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:17.175819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:38819 (local address 127.3.214.194:42017)
0924 16:09:17.176177 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:17.176183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:17.176356 (+ 173us) server_negotiation.cc:407] Connection header received
0924 16:09:17.176602 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:17.176607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:17.176696 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:17.176788 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:09:17.177292 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:17.178027 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:17.180382 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:17.181272 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:17.181436 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:17.181513 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:17.181880 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 16:09:17.182024 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260924 16:09:17.182497 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:17.175434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:42017 (local address 127.3.214.193:38819)
0924 16:09:17.176219 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:17.176259 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:17.176445 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:17.176822 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:17.176826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:17.176856 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:09:17.177142 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:17.177151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:17.178300 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:17.178307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:17.180167 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:17.180188 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:17.181521 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:17.181526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:17.181688 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:17.181723 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 16:09:17.181824 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 16:09:17.181989 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":1,"thread_start_us":352,"threads_started":1}
I20260924 16:09:17.183795 18324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fc1e4f2c9bf43bbada44ca50d14433a" candidate_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4252c8af4a7745c3b0eef21e54f18827" is_pre_election: true
I20260924 16:09:17.183959 18458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fc1e4f2c9bf43bbada44ca50d14433a" candidate_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "065aa5432a694522acaa7a486a720a31" is_pre_election: true
I20260924 16:09:17.184508 18324 raft_consensus.cc:2454] T 6fc1e4f2c9bf43bbada44ca50d14433a P 4252c8af4a7745c3b0eef21e54f18827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1652cd36cab4e51a07e4d1fb6e361df in term 0.
I20260924 16:09:17.184551 18458 raft_consensus.cc:2454] T 6fc1e4f2c9bf43bbada44ca50d14433a P 065aa5432a694522acaa7a486a720a31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1652cd36cab4e51a07e4d1fb6e361df in term 0.
I20260924 16:09:17.185328 18124 leader_election.cc:302] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 065aa5432a694522acaa7a486a720a31, f1652cd36cab4e51a07e4d1fb6e361df; no voters:
I20260924 16:09:17.185838 18528 raft_consensus.cc:2781] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:09:17.186000 18528 raft_consensus.cc:482] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:09:17.186161 18528 raft_consensus.cc:3037] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:09:17.188441 18528 raft_consensus.cc:504] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } } peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } }
I20260924 16:09:17.189141 18528 leader_election.cc:288] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 1 election: Requested vote from peers 065aa5432a694522acaa7a486a720a31 (127.3.214.195:42317), 4252c8af4a7745c3b0eef21e54f18827 (127.3.214.194:42017)
I20260924 16:09:17.189854 18458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fc1e4f2c9bf43bbada44ca50d14433a" candidate_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "065aa5432a694522acaa7a486a720a31"
I20260924 16:09:17.190135 18458 raft_consensus.cc:3037] T 6fc1e4f2c9bf43bbada44ca50d14433a P 065aa5432a694522acaa7a486a720a31 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:09:17.190407 18324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fc1e4f2c9bf43bbada44ca50d14433a" candidate_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4252c8af4a7745c3b0eef21e54f18827"
I20260924 16:09:17.190659 18324 raft_consensus.cc:3037] T 6fc1e4f2c9bf43bbada44ca50d14433a P 4252c8af4a7745c3b0eef21e54f18827 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:09:17.192610 18458 raft_consensus.cc:2454] T 6fc1e4f2c9bf43bbada44ca50d14433a P 065aa5432a694522acaa7a486a720a31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1652cd36cab4e51a07e4d1fb6e361df in term 1.
I20260924 16:09:17.193135 18124 leader_election.cc:302] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 065aa5432a694522acaa7a486a720a31, f1652cd36cab4e51a07e4d1fb6e361df; no voters:
I20260924 16:09:17.193405 18324 raft_consensus.cc:2454] T 6fc1e4f2c9bf43bbada44ca50d14433a P 4252c8af4a7745c3b0eef21e54f18827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1652cd36cab4e51a07e4d1fb6e361df in term 1.
I20260924 16:09:17.193539 18528 raft_consensus.cc:2781] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:09:17.194216 18528 raft_consensus.cc:686] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df [term 1 LEADER]: Becoming Leader. State: Replica: f1652cd36cab4e51a07e4d1fb6e361df, State: Running, Role: LEADER
I20260924 16:09:17.194835 18528 consensus_queue.cc:237] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } } peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } }
I20260924 16:09:17.200645 18044 catalog_manager.cc:5495] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df reported cstate change: term changed from 0 to 1, leader changed from <none> to f1652cd36cab4e51a07e4d1fb6e361df (127.3.214.193). New cstate: current_term: 1 leader_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } health_report { overall_health: UNKNOWN } } }
I20260924 16:09:17.254663 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:17.248444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.3.214.193:34359)
0924 16:09:17.248653 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:17.248658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:17.248679 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:09:17.248809 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:17.248813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:17.248917 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:17.249067 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:09:17.249530 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:17.250410 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:17.251994 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:17.252480 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:17.252594 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:17.253199 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:09:17.253909 (+ 710us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:09:17.254040 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:17.254279 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 16:09:17.254385 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:09:17.265612 18458 raft_consensus.cc:1261] T 6fc1e4f2c9bf43bbada44ca50d14433a P 065aa5432a694522acaa7a486a720a31 [term 1 FOLLOWER]: Refusing update from remote peer f1652cd36cab4e51a07e4d1fb6e361df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:09:17.265609 18324 raft_consensus.cc:1261] T 6fc1e4f2c9bf43bbada44ca50d14433a P 4252c8af4a7745c3b0eef21e54f18827 [term 1 FOLLOWER]: Refusing update from remote peer f1652cd36cab4e51a07e4d1fb6e361df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:09:17.266904 18533 consensus_queue.cc:1035] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df [LEADER]: Connected to new peer: Peer: permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:09:17.267329 18528 consensus_queue.cc:1035] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df [LEADER]: Connected to new peer: Peer: permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:09:17.280185 18536 mvcc.cc:205] Tried to move back new op lower bound from 7332930180145225728 to 7332930179874856960. Current Snapshot: MvccSnapshot[applied={T|T < 7332930180145225728}]
I20260924 16:09:17.299502 18044 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:09:17.299816 18044 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:09:17.304315 18044 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef: Sending DeleteTablet for 3 replicas of tablet 6fc1e4f2c9bf43bbada44ca50d14433a
I20260924 16:09:17.305605 18169 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fc1e4f2c9bf43bbada44ca50d14433a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:09:17 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:34912
I20260924 16:09:17.305871 18438 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fc1e4f2c9bf43bbada44ca50d14433a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:09:17 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:33296
I20260924 16:09:17.305984 18304 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fc1e4f2c9bf43bbada44ca50d14433a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:09:17 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:51558
I20260924 16:09:17.306885 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 18106
I20260924 16:09:17.338992 18545 tablet_replica.cc:330] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df: stopping tablet replica
I20260924 16:09:17.339718 18546 tablet_replica.cc:330] T 6fc1e4f2c9bf43bbada44ca50d14433a P 065aa5432a694522acaa7a486a720a31: stopping tablet replica
I20260924 16:09:17.340384 18546 raft_consensus.cc:2229] T 6fc1e4f2c9bf43bbada44ca50d14433a P 065aa5432a694522acaa7a486a720a31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:09:17.340392 18547 tablet_replica.cc:330] T 6fc1e4f2c9bf43bbada44ca50d14433a P 4252c8af4a7745c3b0eef21e54f18827: stopping tablet replica
I20260924 16:09:17.340399 18545 raft_consensus.cc:2229] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:09:17.341013 18547 raft_consensus.cc:2229] T 6fc1e4f2c9bf43bbada44ca50d14433a P 4252c8af4a7745c3b0eef21e54f18827 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:09:17.341068 18546 raft_consensus.cc:2258] T 6fc1e4f2c9bf43bbada44ca50d14433a P 065aa5432a694522acaa7a486a720a31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:09:17.341158 18545 raft_consensus.cc:2258] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:09:17.343591 18545 ts_tablet_manager.cc:1905] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:09:17.343581 18546 ts_tablet_manager.cc:1905] T 6fc1e4f2c9bf43bbada44ca50d14433a P 065aa5432a694522acaa7a486a720a31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:09:17.344740 18547 raft_consensus.cc:2258] T 6fc1e4f2c9bf43bbada44ca50d14433a P 4252c8af4a7745c3b0eef21e54f18827 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 24 16:09:17 dist-test-slave-1gc3 krb5kdc[18000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.193@KRBTEST.COM
I20260924 16:09:17.351187 18545 ts_tablet_manager.cc:1918] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:09:17.351163 18546 ts_tablet_manager.cc:1918] T 6fc1e4f2c9bf43bbada44ca50d14433a P 065aa5432a694522acaa7a486a720a31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:09:17.351377 18546 log.cc:1196] T 6fc1e4f2c9bf43bbada44ca50d14433a P 065aa5432a694522acaa7a486a720a31: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/6fc1e4f2c9bf43bbada44ca50d14433a
I20260924 16:09:17.351449 18545 log.cc:1196] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/wal/wals/6fc1e4f2c9bf43bbada44ca50d14433a
I20260924 16:09:17.352005 18545 ts_tablet_manager.cc:1939] T 6fc1e4f2c9bf43bbada44ca50d14433a P f1652cd36cab4e51a07e4d1fb6e361df: Deleting consensus metadata
I20260924 16:09:17.351996 18546 ts_tablet_manager.cc:1939] T 6fc1e4f2c9bf43bbada44ca50d14433a P 065aa5432a694522acaa7a486a720a31: Deleting consensus metadata
I20260924 16:09:17.352846 18547 ts_tablet_manager.cc:1905] T 6fc1e4f2c9bf43bbada44ca50d14433a P 4252c8af4a7745c3b0eef21e54f18827: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:09:17.353989 18029 catalog_manager.cc:4841] TS f1652cd36cab4e51a07e4d1fb6e361df (127.3.214.193:34359): tablet 6fc1e4f2c9bf43bbada44ca50d14433a (table test-table [id=2c832d329cca4260bcc41949d2ea2c26]) successfully deleted
I20260924 16:09:17.354172 18031 catalog_manager.cc:4841] TS 065aa5432a694522acaa7a486a720a31 (127.3.214.195:42317): tablet 6fc1e4f2c9bf43bbada44ca50d14433a (table test-table [id=2c832d329cca4260bcc41949d2ea2c26]) successfully deleted
I20260924 16:09:17.361786 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:17.308323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34926 (local address 127.3.214.193:34359)
0924 16:09:17.308549 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:17.308556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:17.330709 (+ 22153us) server_negotiation.cc:407] Connection header received
0924 16:09:17.331362 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:17.331369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:17.331494 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:17.331575 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:17.333986 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:17.335429 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:17.341502 (+ 6073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:17.342438 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:17.342588 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:17.355470 (+ 12882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:17.355483 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:17.355499 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:17.355548 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:17.358729 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:17.359516 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:17.359523 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:17.359526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:17.359682 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:17.360083 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:17.360087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:17.360089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:17.360504 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:17.360791 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:17.361133 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 16:09:17.361400 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 16:09:17.362047 18547 ts_tablet_manager.cc:1918] T 6fc1e4f2c9bf43bbada44ca50d14433a P 4252c8af4a7745c3b0eef21e54f18827: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:09:17.362321 18547 log.cc:1196] T 6fc1e4f2c9bf43bbada44ca50d14433a P 4252c8af4a7745c3b0eef21e54f18827: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/6fc1e4f2c9bf43bbada44ca50d14433a
I20260924 16:09:17.362673 18229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:17.362993 18547 ts_tablet_manager.cc:1939] T 6fc1e4f2c9bf43bbada44ca50d14433a P 4252c8af4a7745c3b0eef21e54f18827: Deleting consensus metadata
I20260924 16:09:17.365494 18031 catalog_manager.cc:4841] TS 4252c8af4a7745c3b0eef21e54f18827 (127.3.214.194:42017): tablet 6fc1e4f2c9bf43bbada44ca50d14433a (table test-table [id=2c832d329cca4260bcc41949d2ea2c26]) successfully deleted
I20260924 16:09:17.470297 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 18106
I20260924 16:09:17.488610 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 18240
Sep 24 16:09:17 dist-test-slave-1gc3 krb5kdc[18000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.194@KRBTEST.COM
I20260924 16:09:17.511677 18521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:17.489576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51564 (local address 127.3.214.194:42017)
0924 16:09:17.489828 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:17.489835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:17.489882 (+ 47us) server_negotiation.cc:407] Connection header received
0924 16:09:17.490017 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:17.490022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:17.490156 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:17.490258 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:17.491790 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:17.492588 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:17.494054 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:17.494588 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:17.494713 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:17.504813 (+ 10100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:17.504846 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:17.504871 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:17.504926 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:17.508371 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:17.509245 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:17.509254 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:17.509257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:17.509412 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:17.509811 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:17.509815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:17.509817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:17.510352 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:17.510690 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:17.511010 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 16:09:17.511280 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 16:09:17.512589 18364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:17.605491 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 18240
I20260924 16:09:17.625761 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 18375
Sep 24 16:09:17 dist-test-slave-1gc3 krb5kdc[18000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.195@KRBTEST.COM
I20260924 16:09:17.644378 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:17.626835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33310 (local address 127.3.214.195:42317)
0924 16:09:17.627079 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:17.627086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:17.627110 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:09:17.627356 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:17.627364 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:17.627459 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:17.627569 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:17.628974 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:17.629869 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:17.631379 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:17.631945 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:17.632080 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:17.637047 (+ 4967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:17.637074 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:17.637101 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:17.637167 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:17.640774 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:17.641802 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:17.641828 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:17.641831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:17.642027 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:17.642511 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:17.642516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:17.642519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:17.643065 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:17.643369 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:17.643677 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 16:09:17.643949 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 16:09:17.645315 18498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:17.750221 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 18375
I20260924 16:09:17.770797 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 18013
I20260924 16:09:17.772336 18075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:17.890486 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 18013
I20260924 16:09:17.912433 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:38473
--webserver_interface=127.3.214.254
--webserver_port=44757
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:38473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:09:18.330051 18553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:18.330495 18553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:18.330642 18553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:18.339010 18553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:09:18.339180 18553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:18.339259 18553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:09:18.339344 18553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:09:18.339411 18553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:09:18.351075 18553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:38473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:38473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=44757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:18.352985 18553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:18.354820 18553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:18.383970 18559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:18.384037 18562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:18.384059 18560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:18.384579 18553 server_base.cc:1034] running on GCE node
I20260924 16:09:18.385987 18553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:18.387944 18553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:09:18.389197 18553 hybrid_clock.cc:648] HybridClock initialized: now 1790266158389110 us; error 79 us; skew 500 ppm
I20260924 16:09:18.389707 18553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:18 dist-test-slave-1gc3 krb5kdc[18000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266158, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:18.399122 18553 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:09:18.401324 18553 webserver.cc:460] Webserver started at http://127.3.214.254:44757/ using document root <none> and password file <none>
I20260924 16:09:18.402282 18553 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:18.402431 18553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:09:18.409933 18553 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:09:18.413925 18569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:18.415861 18553 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:09:18.416119 18553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "d4533be9c9df4f0584500746c806dcef"
format_stamp: "Formatted at 2026-09-24 16:09:14 on dist-test-slave-1gc3"
server_key: "1ce6ed245d758f00a43ccca33cae1e08"
server_key_iv: "2b5ba859e725d70a2e1a764552ad3737"
server_key_version: "encryptionkey@0"
I20260924 16:09:18.417354 18553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:18.439431 18553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:18.444236 18553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:18.444519 18553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:18.467849 18553 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:38473
I20260924 16:09:18.467851 18620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:38473 every 8 connection(s)
I20260924 16:09:18.469620 18553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:09:18.476881 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 18553
I20260924 16:09:18.477730 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:34359
--local_ip_for_outbound_sockets=127.3.214.193
--tserver_master_addrs=127.3.214.254:38473
--webserver_port=37543
--webserver_interface=127.3.214.193
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:09:18.479151 18621 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:09:18.483381 18621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef: Bootstrap starting.
I20260924 16:09:18.514740 18621 log.cc:824] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef: Log is configured to *not* fsync() on all Append() calls
I20260924 16:09:18.535583 18621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:09:18.536464 18621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef: Bootstrap complete.
I20260924 16:09:18.551960 18621 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4533be9c9df4f0584500746c806dcef" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38473 } }
I20260924 16:09:18.553190 18621 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4533be9c9df4f0584500746c806dcef, State: Initialized, Role: FOLLOWER
I20260924 16:09:18.553960 18621 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4533be9c9df4f0584500746c806dcef" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38473 } }
I20260924 16:09:18.554292 18621 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:09:18.554415 18621 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:09:18.554592 18621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:09:18.559337 18621 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4533be9c9df4f0584500746c806dcef" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38473 } }
I20260924 16:09:18.560009 18621 leader_election.cc:302] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4533be9c9df4f0584500746c806dcef; no voters:
I20260924 16:09:18.560768 18621 leader_election.cc:288] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 16:09:18.561154 18625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 2 FOLLOWER]: Leader election won for term 2
I20260924 16:09:18.562348 18625 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 2 LEADER]: Becoming Leader. State: Replica: d4533be9c9df4f0584500746c806dcef, State: Running, Role: LEADER
I20260924 16:09:18.563391 18621 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:09:18.563194 18625 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4533be9c9df4f0584500746c806dcef" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38473 } }
I20260924 16:09:18.566478 18627 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4533be9c9df4f0584500746c806dcef. Latest consensus state: current_term: 2 leader_uuid: "d4533be9c9df4f0584500746c806dcef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4533be9c9df4f0584500746c806dcef" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38473 } } }
I20260924 16:09:18.566938 18627 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [sys.catalog]: This master's current role is: LEADER
I20260924 16:09:18.567610 18626 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d4533be9c9df4f0584500746c806dcef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4533be9c9df4f0584500746c806dcef" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38473 } } }
I20260924 16:09:18.567906 18626 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [sys.catalog]: This master's current role is: LEADER
I20260924 16:09:18.584228 18632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:09:18.601300 18632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:09:18.603127 18632 catalog_manager.cc:1259] Loaded cluster ID: c59b21c5970c42ad80cc9fb7e03a9006
I20260924 16:09:18.614389 18632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:09:18.617470 18632 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 16:09:18.629050 18632 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 16:09:18.636772 18632 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:09:18.638365 18632 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef: Loaded TSK: 0
I20260924 16:09:18.639693 18632 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:09:18.968161 18623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:18.968472 18623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:18.968575 18623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:18.976548 18623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:18.976686 18623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:09:18.976827 18623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:09:18.987504 18623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:34359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=37543
--tserver_master_addrs=127.3.214.254:38473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:18.988987 18623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:18.990486 18623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:19.007296 18649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:19.007296 18650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:19.007493 18652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:19.009593 18623 server_base.cc:1034] running on GCE node
I20260924 16:09:19.010424 18623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:19.011619 18623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:09:19.012873 18623 hybrid_clock.cc:648] HybridClock initialized: now 1790266159012832 us; error 38 us; skew 500 ppm
I20260924 16:09:19.013252 18623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:19 dist-test-slave-1gc3 krb5kdc[18000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266159, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:19.017088 18623 init.cc:377] Logged in from keytab as kudu/127.3.214.193@KRBTEST.COM (short username kudu)
I20260924 16:09:19.019037 18623 webserver.cc:460] Webserver started at http://127.3.214.193:37543/ using document root <none> and password file <none>
I20260924 16:09:19.019887 18623 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:19.020037 18623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:09:19.026397 18623 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:09:19.029491 18659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:19.030994 18623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:09:19.031154 18623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "f1652cd36cab4e51a07e4d1fb6e361df"
format_stamp: "Formatted at 2026-09-24 16:09:15 on dist-test-slave-1gc3"
server_key: "ef75e92240e9f99c8fef941bf343a39b"
server_key_iv: "07ad02358a7722f95df2ca8b7b1309c2"
server_key_version: "encryptionkey@0"
I20260924 16:09:19.032164 18623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:19.050043 18623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:19.055531 18623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:19.055814 18623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:19.057152 18623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:09:19.058776 18623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:09:19.058878 18623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:19.058996 18623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:09:19.059077 18623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:19.095469 18623 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:34359
I20260924 16:09:19.095587 18774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:34359 every 8 connection(s)
I20260924 16:09:19.097147 18623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:09:19.107508 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 18623
I20260924 16:09:19.108155 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:42017
--local_ip_for_outbound_sockets=127.3.214.194
--tserver_master_addrs=127.3.214.254:38473
--webserver_port=35369
--webserver_interface=127.3.214.194
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:09:19 dist-test-slave-1gc3 krb5kdc[18000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266159, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:09:19.124341 18777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:19.100800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:54711 (local address 127.3.214.254:38473)
0924 16:09:19.101519 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:19.101538 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:19.102231 (+ 693us) server_negotiation.cc:407] Connection header received
0924 16:09:19.104882 (+ 2651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:19.104908 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:19.105401 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:19.105777 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:19.107815 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:19.109244 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:19.111021 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:19.111725 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:19.111930 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:19.115952 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:19.115963 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:19.115991 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:19.116045 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:19.119161 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:19.120031 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:19.120040 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:19.120046 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:19.120212 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:19.120658 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:19.120662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:19.120665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:19.121233 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:19.121482 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:19.122370 (+ 888us) server_negotiation.cc:299] Negotiation successful
0924 16:09:19.122828 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":288,"threads_started":1}
I20260924 16:09:19.124370 18778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:19.100985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.193:54711)
0924 16:09:19.101968 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:19.102028 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:19.104157 (+ 2129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:19.106021 (+ 1864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:19.106030 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:19.106734 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:19.107531 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:19.107552 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:19.109437 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:19.109441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:19.110861 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:19.110876 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:19.112006 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:19.112014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:19.112305 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:19.113065 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:19.113097 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:19.115627 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:19.119340 (+ 3713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:19.119352 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:19.119373 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:19.119877 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:19.120328 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:19.120333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:19.120337 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:19.120539 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:19.121377 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:19.121386 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:19.121793 (+ 407us) client_negotiation.cc:769] Sending connection context
0924 16:09:19.122074 (+ 281us) client_negotiation.cc:240] Negotiation successful
0924 16:09:19.122802 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":382,"threads_started":1}
I20260924 16:09:19.128901 18775 heartbeater.cc:347] Connected to a master server at 127.3.214.254:38473
I20260924 16:09:19.129277 18775 heartbeater.cc:464] Registering TS with master...
I20260924 16:09:19.130297 18775 heartbeater.cc:511] Master 127.3.214.254:38473 requested a full tablet report, sending...
I20260924 16:09:19.133224 18586 ts_manager.cc:194] Registered new tserver with Master: f1652cd36cab4e51a07e4d1fb6e361df (127.3.214.193:34359)
I20260924 16:09:19.135074 18586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:54711
W20260924 16:09:19.479462 18779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:19.479856 18779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:19.480033 18779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:19.488868 18779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:19.489027 18779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:09:19.489199 18779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:09:19.500874 18779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:42017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=35369
--tserver_master_addrs=127.3.214.254:38473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:19.502879 18779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:19.504787 18779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:19.520658 18784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:19.520584 18785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:19.520679 18787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:19.521389 18779 server_base.cc:1034] running on GCE node
I20260924 16:09:19.522382 18779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:19.523643 18779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:09:19.524937 18779 hybrid_clock.cc:648] HybridClock initialized: now 1790266159524813 us; error 70 us; skew 500 ppm
I20260924 16:09:19.525344 18779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:19 dist-test-slave-1gc3 krb5kdc[18000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266159, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:19.529341 18779 init.cc:377] Logged in from keytab as kudu/127.3.214.194@KRBTEST.COM (short username kudu)
I20260924 16:09:19.531265 18779 webserver.cc:460] Webserver started at http://127.3.214.194:35369/ using document root <none> and password file <none>
I20260924 16:09:19.532080 18779 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:19.532229 18779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:09:19.538815 18779 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 16:09:19.542549 18794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:19.544611 18779 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:09:19.544849 18779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "4252c8af4a7745c3b0eef21e54f18827"
format_stamp: "Formatted at 2026-09-24 16:09:16 on dist-test-slave-1gc3"
server_key: "1ec67842ee8569017b0974650e36392f"
server_key_iv: "f0e5a6084269af7b6754a4d0f27ecce6"
server_key_version: "encryptionkey@0"
I20260924 16:09:19.545991 18779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:19.567834 18779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:19.572388 18779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:19.572679 18779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:19.573912 18779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:09:19.575593 18779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:09:19.575706 18779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:19.575825 18779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:09:19.575905 18779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:19.612653 18779 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:42017
I20260924 16:09:19.612763 18909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:42017 every 8 connection(s)
I20260924 16:09:19.614262 18779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:09:19.616490 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 18779
I20260924 16:09:19.617318 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:42317
--local_ip_for_outbound_sockets=127.3.214.195
--tserver_master_addrs=127.3.214.254:38473
--webserver_port=35891
--webserver_interface=127.3.214.195
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:09:19 dist-test-slave-1gc3 krb5kdc[18000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266159, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:09:19.640679 18777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:19.617873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:59795 (local address 127.3.214.254:38473)
0924 16:09:19.618201 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:19.618208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:19.619573 (+ 1365us) server_negotiation.cc:407] Connection header received
0924 16:09:19.621602 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:19.621611 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:19.621783 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:19.621963 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:19.623770 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:19.624813 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:19.626738 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:19.627387 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:19.627560 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:19.632857 (+ 5297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:19.632868 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:19.632874 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:19.632934 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:19.636372 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:19.637557 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:19.637564 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:19.637567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:19.637779 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:19.638377 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:19.638382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:19.638384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:19.638831 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:19.638987 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:19.639985 (+ 998us) server_negotiation.cc:299] Negotiation successful
0924 16:09:19.640299 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260924 16:09:19.641940 18912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:19.618434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.194:59795)
0924 16:09:19.619346 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:19.619406 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:19.621260 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:19.622258 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:19.622267 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:19.622878 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:19.623547 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:19.623567 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:19.625073 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:19.625079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:19.626537 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:19.626553 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:19.627744 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:19.627752 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:19.628068 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:19.629267 (+ 1199us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:19.629310 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:19.632397 (+ 3087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:19.636657 (+ 4260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:19.636674 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:19.636695 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:19.637353 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:19.637927 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:19.637931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:19.637934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:19.638236 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:19.639000 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:19.639009 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:19.639517 (+ 508us) client_negotiation.cc:769] Sending connection context
0924 16:09:19.639830 (+ 313us) client_negotiation.cc:240] Negotiation successful
0924 16:09:19.640405 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":362,"threads_started":1}
I20260924 16:09:19.644057 18910 heartbeater.cc:347] Connected to a master server at 127.3.214.254:38473
I20260924 16:09:19.644454 18910 heartbeater.cc:464] Registering TS with master...
I20260924 16:09:19.645466 18910 heartbeater.cc:511] Master 127.3.214.254:38473 requested a full tablet report, sending...
I20260924 16:09:19.647588 18586 ts_manager.cc:194] Registered new tserver with Master: 4252c8af4a7745c3b0eef21e54f18827 (127.3.214.194:42017)
I20260924 16:09:19.648579 18586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:59795
W20260924 16:09:19.983544 18913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:19.983913 18913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:19.984050 18913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:19.991828 18913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:19.991974 18913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:09:19.992132 18913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:09:20.002833 18913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:42317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=35891
--tserver_master_addrs=127.3.214.254:38473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:20.004380 18913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:20.006244 18913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:20.022971 18918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:20.023056 18919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:20.023073 18921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:20.024690 18913 server_base.cc:1034] running on GCE node
I20260924 16:09:20.025468 18913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:20.026759 18913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:09:20.028028 18913 hybrid_clock.cc:648] HybridClock initialized: now 1790266160027969 us; error 46 us; skew 500 ppm
I20260924 16:09:20.028394 18913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:20 dist-test-slave-1gc3 krb5kdc[18000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266160, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:20.031966 18913 init.cc:377] Logged in from keytab as kudu/127.3.214.195@KRBTEST.COM (short username kudu)
I20260924 16:09:20.033752 18913 webserver.cc:460] Webserver started at http://127.3.214.195:35891/ using document root <none> and password file <none>
I20260924 16:09:20.034569 18913 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:20.034704 18913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:09:20.041754 18913 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.002s
I20260924 16:09:20.045768 18928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:20.048151 18913 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.003s
I20260924 16:09:20.048395 18913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "065aa5432a694522acaa7a486a720a31"
format_stamp: "Formatted at 2026-09-24 16:09:16 on dist-test-slave-1gc3"
server_key: "13e70b2a461cc85d44c301125c5735b0"
server_key_iv: "3f66d4de4d6fa2817e9cbacab03ae81e"
server_key_version: "encryptionkey@0"
I20260924 16:09:20.049543 18913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:20.096115 18913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:20.101524 18913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:20.101830 18913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:20.103055 18913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:09:20.104679 18913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:09:20.104789 18913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:20.104915 18913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:09:20.104984 18913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:20.139801 18775 heartbeater.cc:503] Master 127.3.214.254:38473 was elected leader, sending a full tablet report...
I20260924 16:09:20.141436 18913 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:42317
I20260924 16:09:20.141573 19042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:42317 every 8 connection(s)
I20260924 16:09:20.143200 18913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:09:20.153314 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 18913
Sep 24 16:09:20 dist-test-slave-1gc3 krb5kdc[18000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266160, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:09:20.169745 19045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.146664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:60517 (local address 127.3.214.254:38473)
0924 16:09:20.147201 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:20.147206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:20.148309 (+ 1103us) server_negotiation.cc:407] Connection header received
0924 16:09:20.150906 (+ 2597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:20.150914 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:20.151030 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:20.151165 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:20.153329 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.154272 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.156736 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.157235 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.157325 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:20.161870 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:20.161876 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:20.161880 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:20.161922 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:20.165607 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:20.166723 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:20.166729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:20.166735 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:20.166916 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:20.167465 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:20.167470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:20.167473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:20.167832 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:20.167977 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:20.169015 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0924 16:09:20.169309 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":273,"threads_started":1}
I20260924 16:09:20.170661 19046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.147109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.195:60517)
0924 16:09:20.148086 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:20.148146 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:20.150462 (+ 2316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:20.151593 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:20.151604 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:20.152301 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:20.153118 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:20.153140 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.154542 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.154551 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:20.156473 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:20.156498 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.157661 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.157666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:20.157897 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:20.158632 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:20.158660 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:20.161598 (+ 2938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:20.165794 (+ 4196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:20.165806 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:20.165829 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:20.166568 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:20.167050 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:20.167055 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:20.167059 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:20.167302 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:20.167992 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:20.168002 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:20.168517 (+ 515us) client_negotiation.cc:769] Sending connection context
0924 16:09:20.168857 (+ 340us) client_negotiation.cc:240] Negotiation successful
0924 16:09:20.169331 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":367,"threads_started":1}
I20260924 16:09:20.172945 19043 heartbeater.cc:347] Connected to a master server at 127.3.214.254:38473
I20260924 16:09:20.173013 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.154481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44368 (local address 127.3.214.254:38473)
0924 16:09:20.155098 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:20.155102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:20.155704 (+ 602us) server_negotiation.cc:407] Connection header received
0924 16:09:20.155903 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:20.155908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:20.155983 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:20.156079 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:20.157905 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.158801 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.160388 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.161120 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.161299 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:20.165180 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:20.165194 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:20.165198 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:20.165265 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:20.168716 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:20.169638 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:20.169645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:20.169649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:20.169830 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:20.170555 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:20.170561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:20.170563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:20.170959 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:20.171239 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:20.172499 (+ 1260us) server_negotiation.cc:299] Negotiation successful
0924 16:09:20.172709 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":321,"threads_started":1}
I20260924 16:09:20.173369 19043 heartbeater.cc:464] Registering TS with master...
I20260924 16:09:20.174489 19043 heartbeater.cc:511] Master 127.3.214.254:38473 requested a full tablet report, sending...
I20260924 16:09:20.176437 18586 ts_manager.cc:194] Registered new tserver with Master: 065aa5432a694522acaa7a486a720a31 (127.3.214.195:42317)
I20260924 16:09:20.177331 18586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:60517
I20260924 16:09:20.189487 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:09:20.197355 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.190779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44376 (local address 127.3.214.254:38473)
0924 16:09:20.191023 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:20.191030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:20.191531 (+ 501us) server_negotiation.cc:407] Connection header received
0924 16:09:20.191793 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:20.191798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:20.191971 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:20.192122 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:09:20.192613 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.193352 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.194968 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.195413 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.195523 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:20.196082 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:09:20.196699 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:09:20.196842 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:20.196959 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 16:09:20.197056 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 16:09:20.198699 18586 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:09:20.206777 18586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:09:20.234776 19054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.222733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:42017 (local address 127.0.0.1:51576)
0924 16:09:20.223955 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:20.224027 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:20.224245 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:20.226271 (+ 2026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:20.226279 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:20.226313 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:09:20.226773 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:20.226788 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.228853 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.228860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:20.231720 (+ 2860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:20.231741 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.233412 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.233428 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:20.233689 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:20.233766 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 16:09:20.233930 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 16:09:20.234221 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":1,"thread_start_us":685,"threads_started":1}
I20260924 16:09:20.236059 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.222847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.3.214.194:42017)
0924 16:09:20.225566 (+ 2719us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:20.225573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:20.225600 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:09:20.225716 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:20.225725 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:20.225997 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:20.226256 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:09:20.226990 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.228683 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.231901 (+ 3218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.233133 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.233296 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:20.233533 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:20.235358 (+ 1825us) server_negotiation.cc:299] Negotiation successful
0924 16:09:20.235626 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2566,"thread_start_us":2436,"threads_started":1}
I20260924 16:09:20.237679 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.223646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34942 (local address 127.3.214.193:34359)
0924 16:09:20.227936 (+ 4290us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:20.227946 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:20.227985 (+ 39us) server_negotiation.cc:407] Connection header received
0924 16:09:20.228105 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:20.228115 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:20.228569 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:20.229535 (+ 966us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:09:20.229834 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.231628 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.234180 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.235942 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.236166 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:20.236449 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:20.237153 (+ 704us) server_negotiation.cc:299] Negotiation successful
0924 16:09:20.237340 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4080,"thread_start_us":3892,"threads_started":1}
I20260924 16:09:20.238034 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.222837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:34359 (local address 127.0.0.1:34942)
0924 16:09:20.224861 (+ 2024us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:20.224886 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:20.225048 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:20.228858 (+ 3810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:20.228862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:20.228891 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:09:20.229299 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:20.229313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.231817 (+ 2504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.231823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:20.233907 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:20.233931 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.236622 (+ 2691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.236630 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:20.236842 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:20.236889 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 16:09:20.237023 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 16:09:20.237216 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1854,"mutex_wait_us":1,"thread_start_us":1654,"threads_started":1}
I20260924 16:09:20.238502 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.227655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:42317 (local address 127.0.0.1:33312)
0924 16:09:20.228278 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:20.228304 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:20.228444 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:20.229878 (+ 1434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:20.229883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:20.229912 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:09:20.230771 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:20.230785 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.232801 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.232808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:20.235142 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:20.235158 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.237530 (+ 2372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.237536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:20.237714 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:20.237747 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 16:09:20.237819 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 16:09:20.237962 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":76,"thread_start_us":919,"threads_started":1}
I20260924 16:09:20.239117 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.228409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33312 (local address 127.3.214.195:42317)
0924 16:09:20.229216 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:20.229223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:20.229253 (+ 30us) server_negotiation.cc:407] Connection header received
0924 16:09:20.229340 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:20.229347 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:20.229595 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:20.230945 (+ 1350us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:09:20.231191 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.232605 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.235407 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.237242 (+ 1835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.238212 (+ 970us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:20.238507 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:20.238598 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 16:09:20.238801 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":478,"threads_started":1}
I20260924 16:09:20.241243 18843 tablet_service.cc:1467] Processing CreateTablet for tablet 96d4659136634e2bb6a42c4e431efd34 (DEFAULT_TABLE table=test-table [id=c8d5017515134c71a90c8534c67f210e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:09:20.242857 18843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96d4659136634e2bb6a42c4e431efd34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:09:20.243094 18708 tablet_service.cc:1467] Processing CreateTablet for tablet 96d4659136634e2bb6a42c4e431efd34 (DEFAULT_TABLE table=test-table [id=c8d5017515134c71a90c8534c67f210e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:09:20.243094 18976 tablet_service.cc:1467] Processing CreateTablet for tablet 96d4659136634e2bb6a42c4e431efd34 (DEFAULT_TABLE table=test-table [id=c8d5017515134c71a90c8534c67f210e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:09:20.245165 18976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96d4659136634e2bb6a42c4e431efd34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:09:20.245189 18708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96d4659136634e2bb6a42c4e431efd34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:09:20.258457 19060 tablet_bootstrap.cc:492] T 96d4659136634e2bb6a42c4e431efd34 P 4252c8af4a7745c3b0eef21e54f18827: Bootstrap starting.
I20260924 16:09:20.262657 19060 tablet_bootstrap.cc:654] T 96d4659136634e2bb6a42c4e431efd34 P 4252c8af4a7745c3b0eef21e54f18827: Neither blocks nor log segments found. Creating new log.
I20260924 16:09:20.264187 19061 tablet_bootstrap.cc:492] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df: Bootstrap starting.
I20260924 16:09:20.264938 19062 tablet_bootstrap.cc:492] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31: Bootstrap starting.
I20260924 16:09:20.266542 19060 log.cc:824] T 96d4659136634e2bb6a42c4e431efd34 P 4252c8af4a7745c3b0eef21e54f18827: Log is configured to *not* fsync() on all Append() calls
I20260924 16:09:20.269022 19062 tablet_bootstrap.cc:654] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31: Neither blocks nor log segments found. Creating new log.
I20260924 16:09:20.269837 19061 tablet_bootstrap.cc:654] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df: Neither blocks nor log segments found. Creating new log.
I20260924 16:09:20.270505 19062 log.cc:824] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31: Log is configured to *not* fsync() on all Append() calls
I20260924 16:09:20.270869 19060 tablet_bootstrap.cc:492] T 96d4659136634e2bb6a42c4e431efd34 P 4252c8af4a7745c3b0eef21e54f18827: No bootstrap required, opened a new log
I20260924 16:09:20.271144 19061 log.cc:824] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df: Log is configured to *not* fsync() on all Append() calls
I20260924 16:09:20.271256 19060 ts_tablet_manager.cc:1397] T 96d4659136634e2bb6a42c4e431efd34 P 4252c8af4a7745c3b0eef21e54f18827: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260924 16:09:20.274396 19062 tablet_bootstrap.cc:492] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31: No bootstrap required, opened a new log
I20260924 16:09:20.274682 19061 tablet_bootstrap.cc:492] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df: No bootstrap required, opened a new log
I20260924 16:09:20.274770 19062 ts_tablet_manager.cc:1397] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260924 16:09:20.274952 19061 ts_tablet_manager.cc:1397] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260924 16:09:20.280293 19060 raft_consensus.cc:348] T 96d4659136634e2bb6a42c4e431efd34 P 4252c8af4a7745c3b0eef21e54f18827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } } peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } }
I20260924 16:09:20.280815 19060 raft_consensus.cc:374] T 96d4659136634e2bb6a42c4e431efd34 P 4252c8af4a7745c3b0eef21e54f18827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:09:20.280932 19060 raft_consensus.cc:729] T 96d4659136634e2bb6a42c4e431efd34 P 4252c8af4a7745c3b0eef21e54f18827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4252c8af4a7745c3b0eef21e54f18827, State: Initialized, Role: FOLLOWER
I20260924 16:09:20.281661 19060 consensus_queue.cc:260] T 96d4659136634e2bb6a42c4e431efd34 P 4252c8af4a7745c3b0eef21e54f18827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } } peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } }
I20260924 16:09:20.283010 18910 heartbeater.cc:503] Master 127.3.214.254:38473 was elected leader, sending a full tablet report...
I20260924 16:09:20.285408 19060 ts_tablet_manager.cc:1428] T 96d4659136634e2bb6a42c4e431efd34 P 4252c8af4a7745c3b0eef21e54f18827: Time spent starting tablet: real 0.014s user 0.006s sys 0.006s
I20260924 16:09:20.285847 19062 raft_consensus.cc:348] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } } peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } }
I20260924 16:09:20.285847 19061 raft_consensus.cc:348] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } } peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } }
I20260924 16:09:20.286455 19061 raft_consensus.cc:374] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:09:20.286455 19062 raft_consensus.cc:374] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:09:20.286580 19062 raft_consensus.cc:729] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 065aa5432a694522acaa7a486a720a31, State: Initialized, Role: FOLLOWER
I20260924 16:09:20.286587 19061 raft_consensus.cc:729] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1652cd36cab4e51a07e4d1fb6e361df, State: Initialized, Role: FOLLOWER
I20260924 16:09:20.287379 19062 consensus_queue.cc:260] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } } peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } }
I20260924 16:09:20.287376 19061 consensus_queue.cc:260] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } } peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } }
I20260924 16:09:20.289623 19062 ts_tablet_manager.cc:1428] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260924 16:09:20.291725 19043 heartbeater.cc:503] Master 127.3.214.254:38473 was elected leader, sending a full tablet report...
I20260924 16:09:20.292171 19061 ts_tablet_manager.cc:1428] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
I20260924 16:09:20.295022 19068 raft_consensus.cc:482] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:09:20.295405 19068 raft_consensus.cc:504] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } } peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } }
I20260924 16:09:20.297801 19068 leader_election.cc:288] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4252c8af4a7745c3b0eef21e54f18827 (127.3.214.194:42017), f1652cd36cab4e51a07e4d1fb6e361df (127.3.214.193:34359)
I20260924 16:09:20.303817 19046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.298026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:34359 (local address 127.3.214.195:55795)
0924 16:09:20.298327 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:20.298353 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:20.298544 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:20.299130 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:20.299136 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:20.299178 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:09:20.299387 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:20.299397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.300460 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.300467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:20.301834 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:20.301850 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.303050 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.303065 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:20.303263 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:20.303290 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 16:09:20.303377 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 16:09:20.303531 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260924 16:09:20.303820 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.298286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:55795 (local address 127.3.214.193:34359)
0924 16:09:20.298559 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:20.298566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:20.298596 (+ 30us) server_negotiation.cc:407] Connection header received
0924 16:09:20.298694 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:20.298700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:20.298795 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:20.298908 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:09:20.299535 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.300316 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.302017 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.302890 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.303050 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:20.303136 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:20.303420 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 16:09:20.303565 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 16:09:20.305341 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.298052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:42017 (local address 127.3.214.195:56243)
0924 16:09:20.299328 (+ 1276us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:20.299348 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:20.299528 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:20.299858 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:20.299862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:20.299887 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:09:20.300233 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:20.300243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.301298 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.301303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:20.302829 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:20.302842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.304492 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.304499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:20.304673 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:20.304708 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 16:09:20.304823 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 16:09:20.304958 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"mutex_wait_us":181,"thread_start_us":721,"threads_started":1}
I20260924 16:09:20.305508 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.298725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:56243 (local address 127.3.214.194:42017)
0924 16:09:20.298956 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:20.298963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:20.299439 (+ 476us) server_negotiation.cc:407] Connection header received
0924 16:09:20.299673 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:20.299678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:20.299750 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:20.299827 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:09:20.300379 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.301142 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.302991 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.303822 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.304006 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:20.304092 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:20.304954 (+ 862us) server_negotiation.cc:299] Negotiation successful
0924 16:09:20.305199 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:09:20.305553 18728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d4659136634e2bb6a42c4e431efd34" candidate_uuid: "065aa5432a694522acaa7a486a720a31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" is_pre_election: true
I20260924 16:09:20.306237 18728 raft_consensus.cc:2454] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 065aa5432a694522acaa7a486a720a31 in term 0.
I20260924 16:09:20.307091 18929 leader_election.cc:302] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 065aa5432a694522acaa7a486a720a31, f1652cd36cab4e51a07e4d1fb6e361df; no voters:
I20260924 16:09:20.307016 18864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d4659136634e2bb6a42c4e431efd34" candidate_uuid: "065aa5432a694522acaa7a486a720a31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4252c8af4a7745c3b0eef21e54f18827" is_pre_election: true
I20260924 16:09:20.307660 18864 raft_consensus.cc:2454] T 96d4659136634e2bb6a42c4e431efd34 P 4252c8af4a7745c3b0eef21e54f18827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 065aa5432a694522acaa7a486a720a31 in term 0.
I20260924 16:09:20.308178 19068 raft_consensus.cc:2781] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:09:20.308374 19068 raft_consensus.cc:482] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:09:20.308507 19068 raft_consensus.cc:3037] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:09:20.311188 19068 raft_consensus.cc:504] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } } peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } }
I20260924 16:09:20.311956 19068 leader_election.cc:288] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31 [CANDIDATE]: Term 1 election: Requested vote from peers 4252c8af4a7745c3b0eef21e54f18827 (127.3.214.194:42017), f1652cd36cab4e51a07e4d1fb6e361df (127.3.214.193:34359)
I20260924 16:09:20.312768 18864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d4659136634e2bb6a42c4e431efd34" candidate_uuid: "065aa5432a694522acaa7a486a720a31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4252c8af4a7745c3b0eef21e54f18827"
I20260924 16:09:20.312862 18728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d4659136634e2bb6a42c4e431efd34" candidate_uuid: "065aa5432a694522acaa7a486a720a31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1652cd36cab4e51a07e4d1fb6e361df"
I20260924 16:09:20.312995 18864 raft_consensus.cc:3037] T 96d4659136634e2bb6a42c4e431efd34 P 4252c8af4a7745c3b0eef21e54f18827 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:09:20.313148 18728 raft_consensus.cc:3037] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:09:20.315706 18728 raft_consensus.cc:2454] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 065aa5432a694522acaa7a486a720a31 in term 1.
I20260924 16:09:20.315714 18864 raft_consensus.cc:2454] T 96d4659136634e2bb6a42c4e431efd34 P 4252c8af4a7745c3b0eef21e54f18827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 065aa5432a694522acaa7a486a720a31 in term 1.
I20260924 16:09:20.316293 18931 leader_election.cc:302] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 065aa5432a694522acaa7a486a720a31, 4252c8af4a7745c3b0eef21e54f18827; no voters:
I20260924 16:09:20.316650 19068 raft_consensus.cc:2781] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:09:20.317178 19068 raft_consensus.cc:686] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31 [term 1 LEADER]: Becoming Leader. State: Replica: 065aa5432a694522acaa7a486a720a31, State: Running, Role: LEADER
I20260924 16:09:20.317755 19068 consensus_queue.cc:237] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } } peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } }
I20260924 16:09:20.323305 18585 catalog_manager.cc:5495] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31 reported cstate change: term changed from 0 to 1, leader changed from <none> to 065aa5432a694522acaa7a486a720a31 (127.3.214.195). New cstate: current_term: 1 leader_uuid: "065aa5432a694522acaa7a486a720a31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } health_report { overall_health: HEALTHY } } }
W20260924 16:09:20.350428 18776 tablet.cc:2369] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:09:20.366838 18911 tablet.cc:2369] T 96d4659136634e2bb6a42c4e431efd34 P 4252c8af4a7745c3b0eef21e54f18827: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:09:20.367990 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.361117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33314 (local address 127.3.214.195:42317)
0924 16:09:20.361330 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:20.361338 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:20.361420 (+ 82us) server_negotiation.cc:407] Connection header received
0924 16:09:20.361738 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:20.361743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:20.361888 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:20.362078 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:09:20.362567 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.363388 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.365037 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.365490 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.365600 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:20.366214 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:09:20.367222 (+ 1008us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:09:20.367398 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:20.367550 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 16:09:20.367668 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:09:20.378000 18728 raft_consensus.cc:1261] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Refusing update from remote peer 065aa5432a694522acaa7a486a720a31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:09:20.378172 18864 raft_consensus.cc:1261] T 96d4659136634e2bb6a42c4e431efd34 P 4252c8af4a7745c3b0eef21e54f18827 [term 1 FOLLOWER]: Refusing update from remote peer 065aa5432a694522acaa7a486a720a31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:09:20.379380 19068 consensus_queue.cc:1035] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:09:20.379793 19070 consensus_queue.cc:1035] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:09:20.413982 18586 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:09:20.414450 18586 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:09:20.419356 18586 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef: Sending DeleteTablet for 3 replicas of tablet 96d4659136634e2bb6a42c4e431efd34
I20260924 16:09:20.421155 18843 tablet_service.cc:1514] Processing DeleteTablet for tablet 96d4659136634e2bb6a42c4e431efd34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:09:20 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:51576
I20260924 16:09:20.421427 18976 tablet_service.cc:1514] Processing DeleteTablet for tablet 96d4659136634e2bb6a42c4e431efd34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:09:20 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:33312
I20260924 16:09:20.421438 18708 tablet_service.cc:1514] Processing DeleteTablet for tablet 96d4659136634e2bb6a42c4e431efd34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:09:20 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:34942
I20260924 16:09:20.422781 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 18623
I20260924 16:09:20.432554 19083 tablet_replica.cc:330] T 96d4659136634e2bb6a42c4e431efd34 P 4252c8af4a7745c3b0eef21e54f18827: stopping tablet replica
I20260924 16:09:20.432898 19084 tablet_replica.cc:330] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df: stopping tablet replica
I20260924 16:09:20.433568 19084 raft_consensus.cc:2229] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:09:20.433902 19083 raft_consensus.cc:2229] T 96d4659136634e2bb6a42c4e431efd34 P 4252c8af4a7745c3b0eef21e54f18827 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:09:20.434589 19084 raft_consensus.cc:2258] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:09:20.434793 19083 raft_consensus.cc:2258] T 96d4659136634e2bb6a42c4e431efd34 P 4252c8af4a7745c3b0eef21e54f18827 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:09:20.437950 19083 ts_tablet_manager.cc:1905] T 96d4659136634e2bb6a42c4e431efd34 P 4252c8af4a7745c3b0eef21e54f18827: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:09:20.438870 19084 ts_tablet_manager.cc:1905] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:09:20.442746 19085 tablet_replica.cc:330] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31: stopping tablet replica
I20260924 16:09:20.443313 19085 raft_consensus.cc:2229] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:09:20.444106 19085 raft_consensus.cc:2258] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:09:20.446230 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.423917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34958 (local address 127.3.214.193:34359)
0924 16:09:20.424114 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:20.424121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:20.424278 (+ 157us) server_negotiation.cc:407] Connection header received
0924 16:09:20.424493 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:20.424497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:20.424598 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:20.424718 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:20.426707 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.427943 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.430060 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.430840 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.430992 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:20.435554 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:20.435587 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:20.435607 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:20.435669 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:20.440490 (+ 4821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:20.441713 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:20.441725 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:20.441729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:20.441960 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:20.442594 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:20.442600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:20.442604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:20.443246 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:20.443653 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:20.444795 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0924 16:09:20.445135 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 16:09:20.447469 18770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:20.448057 19083 ts_tablet_manager.cc:1918] T 96d4659136634e2bb6a42c4e431efd34 P 4252c8af4a7745c3b0eef21e54f18827: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:09:20.448315 19083 log.cc:1196] T 96d4659136634e2bb6a42c4e431efd34 P 4252c8af4a7745c3b0eef21e54f18827: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/96d4659136634e2bb6a42c4e431efd34
I20260924 16:09:20.448966 19083 ts_tablet_manager.cc:1939] T 96d4659136634e2bb6a42c4e431efd34 P 4252c8af4a7745c3b0eef21e54f18827: Deleting consensus metadata
I20260924 16:09:20.450318 19085 ts_tablet_manager.cc:1905] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:09:20.451635 18572 catalog_manager.cc:4841] TS 4252c8af4a7745c3b0eef21e54f18827 (127.3.214.194:42017): tablet 96d4659136634e2bb6a42c4e431efd34 (table test-table [id=c8d5017515134c71a90c8534c67f210e]) successfully deleted
I20260924 16:09:20.460034 19085 ts_tablet_manager.cc:1918] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:09:20.460345 19085 log.cc:1196] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/96d4659136634e2bb6a42c4e431efd34
I20260924 16:09:20.461046 19085 ts_tablet_manager.cc:1939] T 96d4659136634e2bb6a42c4e431efd34 P 065aa5432a694522acaa7a486a720a31: Deleting consensus metadata
I20260924 16:09:20.463757 18572 catalog_manager.cc:4841] TS 065aa5432a694522acaa7a486a720a31 (127.3.214.195:42317): tablet 96d4659136634e2bb6a42c4e431efd34 (table test-table [id=c8d5017515134c71a90c8534c67f210e]) successfully deleted
I20260924 16:09:20.568323 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 18623
W20260924 16:09:20.589937 18570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.214.193:34359 (error 108)
I20260924 16:09:20.590145 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 18779
I20260924 16:09:20.598893 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.597992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:34359 (local address 127.0.0.1:34970)
0924 16:09:20.598489 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:20.598657 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:34359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326}
W20260924 16:09:20.600718 18570 catalog_manager.cc:4584] TS f1652cd36cab4e51a07e4d1fb6e361df (127.3.214.193:34359): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 96d4659136634e2bb6a42c4e431efd34: Network error: Client connection negotiation failed: client connection to 127.3.214.193:34359: connect: Connection refused (error 111)
I20260924 16:09:20.610739 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.591410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51578 (local address 127.3.214.194:42017)
0924 16:09:20.591784 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:20.591793 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:20.591831 (+ 38us) server_negotiation.cc:407] Connection header received
0924 16:09:20.591974 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:20.591981 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:20.592141 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:20.592586 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:20.595613 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.596932 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.598954 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.599611 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.599755 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:20.603173 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:20.603193 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:20.603216 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:20.603260 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:20.607487 (+ 4227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:20.608306 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:20.608316 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:20.608320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:20.608501 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:20.608901 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:20.608906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:20.608909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:20.609448 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:20.609784 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:20.610057 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 16:09:20.610350 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260924 16:09:20.611624 18905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:20.655485 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.654609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:34359 (local address 127.0.0.1:34974)
0924 16:09:20.654982 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:20.655177 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:34359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260924 16:09:20.657097 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.656517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:34359 (local address 127.0.0.1:34984)
0924 16:09:20.656749 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:20.656905 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:34359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 16:09:20.715032 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 18779
I20260924 16:09:20.718278 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.717396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:34359 (local address 127.0.0.1:34992)
0924 16:09:20.717759 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:20.717939 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:34359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260924 16:09:20.720146 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.719659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:34359 (local address 127.0.0.1:34994)
0924 16:09:20.719842 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:20.719944 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:34359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 16:09:20.738075 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 18913
I20260924 16:09:20.758880 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.739417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33318 (local address 127.3.214.195:42317)
0924 16:09:20.739642 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:20.739647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:20.739673 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:09:20.739894 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:20.739904 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:20.740005 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:20.740118 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:20.742076 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.743170 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.745204 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:20.745747 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:20.745883 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:20.749898 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:20.749923 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:20.749950 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:20.750016 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:20.754538 (+ 4522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:20.755794 (+ 1256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:20.755808 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:20.755810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:20.756109 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:20.756689 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:20.756692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:20.756694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:20.757218 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:20.757587 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:20.757964 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 16:09:20.758490 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 16:09:20.759862 19038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:20.801623 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.800874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:34359 (local address 127.0.0.1:34998)
0924 16:09:20.801237 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:20.801392 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:34359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260924 16:09:20.803028 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:20.802526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:34359 (local address 127.0.0.1:35004)
0924 16:09:20.802786 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:20.802887 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:34359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260924 16:09:20.896059 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 18913
I20260924 16:09:20.917033 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 18553
I20260924 16:09:20.918336 18616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:21.040742 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 18553
I20260924 16:09:21.065874 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:38473
--webserver_interface=127.3.214.254
--webserver_port=44757
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:38473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:09:21.419390 19091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:21.419755 19091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:21.419899 19091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:21.428750 19091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:09:21.428951 19091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:21.429051 19091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:09:21.429154 19091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:09:21.429230 19091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:09:21.441107 19091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:38473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:38473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=44757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:21.442999 19091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:21.444693 19091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:21.459547 19096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:21.459607 19097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:21.459429 19099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:21.464005 19091 server_base.cc:1034] running on GCE node
I20260924 16:09:21.465054 19091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:21.466958 19091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:09:21.468233 19091 hybrid_clock.cc:648] HybridClock initialized: now 1790266161468168 us; error 62 us; skew 500 ppm
I20260924 16:09:21.468659 19091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:21 dist-test-slave-1gc3 krb5kdc[18000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266161, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:21.480614 19091 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:09:21.482735 19091 webserver.cc:460] Webserver started at http://127.3.214.254:44757/ using document root <none> and password file <none>
I20260924 16:09:21.483523 19091 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:21.483666 19091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:09:21.490375 19091 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 16:09:21.495882 19106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:21.498064 19091 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260924 16:09:21.498373 19091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "d4533be9c9df4f0584500746c806dcef"
format_stamp: "Formatted at 2026-09-24 16:09:14 on dist-test-slave-1gc3"
server_key: "1ce6ed245d758f00a43ccca33cae1e08"
server_key_iv: "2b5ba859e725d70a2e1a764552ad3737"
server_key_version: "encryptionkey@0"
I20260924 16:09:21.499608 19091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:21.523792 19091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:21.528347 19091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:21.528612 19091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:21.549376 19091 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:38473
I20260924 16:09:21.549424 19157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:38473 every 8 connection(s)
I20260924 16:09:21.551110 19091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:09:21.557953 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 19091
I20260924 16:09:21.559171 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:34359
--local_ip_for_outbound_sockets=127.3.214.193
--tserver_master_addrs=127.3.214.254:38473
--webserver_port=37543
--webserver_interface=127.3.214.193
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:09:21.560511 19158 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:09:21.565721 19158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef: Bootstrap starting.
I20260924 16:09:21.592998 19158 log.cc:824] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef: Log is configured to *not* fsync() on all Append() calls
I20260924 16:09:21.612648 19158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:09:21.613317 19158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef: Bootstrap complete.
I20260924 16:09:21.624289 19158 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4533be9c9df4f0584500746c806dcef" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38473 } }
I20260924 16:09:21.625586 19158 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4533be9c9df4f0584500746c806dcef, State: Initialized, Role: FOLLOWER
I20260924 16:09:21.626494 19158 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4533be9c9df4f0584500746c806dcef" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38473 } }
I20260924 16:09:21.626767 19158 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:09:21.626919 19158 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:09:21.627110 19158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 2 FOLLOWER]: Advancing to term 3
I20260924 16:09:21.631994 19158 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4533be9c9df4f0584500746c806dcef" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38473 } }
I20260924 16:09:21.632633 19158 leader_election.cc:302] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4533be9c9df4f0584500746c806dcef; no voters:
I20260924 16:09:21.633349 19158 leader_election.cc:288] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 16:09:21.633523 19162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 3 FOLLOWER]: Leader election won for term 3
I20260924 16:09:21.634961 19162 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [term 3 LEADER]: Becoming Leader. State: Replica: d4533be9c9df4f0584500746c806dcef, State: Running, Role: LEADER
I20260924 16:09:21.635711 19162 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4533be9c9df4f0584500746c806dcef" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38473 } }
I20260924 16:09:21.636210 19158 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:09:21.639065 19163 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d4533be9c9df4f0584500746c806dcef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4533be9c9df4f0584500746c806dcef" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38473 } } }
I20260924 16:09:21.638999 19164 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4533be9c9df4f0584500746c806dcef. Latest consensus state: current_term: 3 leader_uuid: "d4533be9c9df4f0584500746c806dcef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4533be9c9df4f0584500746c806dcef" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 38473 } } }
I20260924 16:09:21.639596 19163 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [sys.catalog]: This master's current role is: LEADER
I20260924 16:09:21.639629 19164 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4533be9c9df4f0584500746c806dcef [sys.catalog]: This master's current role is: LEADER
I20260924 16:09:21.647074 19170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:09:21.657845 19170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:09:21.659662 19170 catalog_manager.cc:1259] Loaded cluster ID: c59b21c5970c42ad80cc9fb7e03a9006
I20260924 16:09:21.659864 19170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 16:09:21.662324 19170 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb448ecb9ad google::LogMessage::Fail() at ??:0
@ 0x7fb448ece88a google::LogMessage::SendToLog() at ??:0
@ 0x7fb448ecb4de google::LogMessage::Flush() at ??:0
@ 0x7fb448ecca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb45e8fe512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb45e8fb976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb45e983610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb45e983439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb44bb17345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb44bb1a91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb44bb1a799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb44baf5eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb44edcf6db start_thread at ??:0
@ 0x7fb447d6561f clone at ??:0
W20260924 16:09:21.974377 19160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:21.974717 19160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:21.974860 19160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:21.982659 19160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:21.982806 19160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:09:21.982976 19160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:09:21.993881 19160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:34359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=37543
--tserver_master_addrs=127.3.214.254:38473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:21.995316 19160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:21.996773 19160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:22.012287 19200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:22.012364 19203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:22.012347 19201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:22.013732 19160 server_base.cc:1034] running on GCE node
I20260924 16:09:22.014685 19160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:22.015970 19160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:09:22.017416 19160 hybrid_clock.cc:648] HybridClock initialized: now 1790266162017339 us; error 61 us; skew 500 ppm
I20260924 16:09:22.017827 19160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:22 dist-test-slave-1gc3 krb5kdc[18000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:22.021658 19160 init.cc:377] Logged in from keytab as kudu/127.3.214.193@KRBTEST.COM (short username kudu)
I20260924 16:09:22.023631 19160 webserver.cc:460] Webserver started at http://127.3.214.193:37543/ using document root <none> and password file <none>
I20260924 16:09:22.024530 19160 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:22.024649 19160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:09:22.031563 19160 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 16:09:22.035524 19210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:22.037516 19160 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:09:22.037765 19160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "f1652cd36cab4e51a07e4d1fb6e361df"
format_stamp: "Formatted at 2026-09-24 16:09:15 on dist-test-slave-1gc3"
server_key: "ef75e92240e9f99c8fef941bf343a39b"
server_key_iv: "07ad02358a7722f95df2ca8b7b1309c2"
server_key_version: "encryptionkey@0"
I20260924 16:09:22.039074 19160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:22.057735 19160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:22.063109 19160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:22.063412 19160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:22.064692 19160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:09:22.067209 19217 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 16:09:22.073848 19160 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 16:09:22.074086 19160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260924 16:09:22.074383 19160 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 16:09:22.078867 19160 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 16:09:22.078974 19160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.002s
I20260924 16:09:22.079556 19217 tablet_bootstrap.cc:492] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df: Bootstrap starting.
I20260924 16:09:22.089133 19217 log.cc:824] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df: Log is configured to *not* fsync() on all Append() calls
I20260924 16:09:22.105222 19217 tablet_bootstrap.cc:492] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:09:22.106256 19217 tablet_bootstrap.cc:492] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df: Bootstrap complete.
I20260924 16:09:22.106755 19217 ts_tablet_manager.cc:1397] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df: Time spent bootstrapping tablet: real 0.028s user 0.026s sys 0.000s
I20260924 16:09:22.115967 19217 raft_consensus.cc:348] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } } peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } }
I20260924 16:09:22.117147 19217 raft_consensus.cc:729] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1652cd36cab4e51a07e4d1fb6e361df, State: Initialized, Role: FOLLOWER
I20260924 16:09:22.118247 19217 consensus_queue.cc:260] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } } peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } }
I20260924 16:09:22.120132 19217 ts_tablet_manager.cc:1428] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260924 16:09:22.129719 19160 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:34359
I20260924 16:09:22.129781 19326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:34359 every 8 connection(s)
I20260924 16:09:22.131407 19160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:09:22.137013 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 19160
I20260924 16:09:22.137391 19329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:22.134818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.193:59087)
0924 16:09:22.135860 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:22.136168 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":409,"threads_started":1}
I20260924 16:09:22.137777 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:42017
--local_ip_for_outbound_sockets=127.3.214.194
--tserver_master_addrs=127.3.214.254:38473
--webserver_port=35369
--webserver_interface=127.3.214.194
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:09:22.138235 19327 heartbeater.cc:644] Failed to heartbeat to 127.3.214.254:38473 (0 consecutive failures): Network error: Failed to ping master at 127.3.214.254:38473: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
I20260924 16:09:22.139175 19329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:22.138819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.193:33519)
0924 16:09:22.138982 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:22.139056 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 16:09:22.140033 19329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:22.139680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.193:56457)
0924 16:09:22.139831 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:22.139904 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 16:09:22.140470 19327 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:09:22.491272 19330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:22.491715 19330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:22.491842 19330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:22.500496 19330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:22.500658 19330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:09:22.500802 19330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:09:22.512233 19330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:42017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=35369
--tserver_master_addrs=127.3.214.254:38473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:22.514181 19330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:22.515967 19330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:22.531915 19336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:22.532174 19335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:22.532215 19338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:22.533365 19330 server_base.cc:1034] running on GCE node
I20260924 16:09:22.534255 19330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:22.535528 19330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:09:22.536777 19330 hybrid_clock.cc:648] HybridClock initialized: now 1790266162536695 us; error 76 us; skew 500 ppm
I20260924 16:09:22.537184 19330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:22 dist-test-slave-1gc3 krb5kdc[18000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:22.541163 19330 init.cc:377] Logged in from keytab as kudu/127.3.214.194@KRBTEST.COM (short username kudu)
I20260924 16:09:22.543267 19330 webserver.cc:460] Webserver started at http://127.3.214.194:35369/ using document root <none> and password file <none>
I20260924 16:09:22.544227 19330 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:22.544363 19330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:09:22.551529 19330 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 16:09:22.555460 19345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:22.557394 19330 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:09:22.557652 19330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "4252c8af4a7745c3b0eef21e54f18827"
format_stamp: "Formatted at 2026-09-24 16:09:16 on dist-test-slave-1gc3"
server_key: "1ec67842ee8569017b0974650e36392f"
server_key_iv: "f0e5a6084269af7b6754a4d0f27ecce6"
server_key_version: "encryptionkey@0"
I20260924 16:09:22.559146 19330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:22.588348 19330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:22.595293 19330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:22.595688 19330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:22.600816 19330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:09:22.603188 19330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:09:22.603350 19330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:22.603499 19330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:09:22.603596 19330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:22.648952 19330 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:42017
I20260924 16:09:22.649068 19457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:42017 every 8 connection(s)
I20260924 16:09:22.650772 19330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:09:22.651155 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 19330
I20260924 16:09:22.651878 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:42317
--local_ip_for_outbound_sockets=127.3.214.195
--tserver_master_addrs=127.3.214.254:38473
--webserver_port=35891
--webserver_interface=127.3.214.195
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:09:22.656852 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:22.654467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.194:39713)
0924 16:09:22.655417 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:22.655760 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":372,"threads_started":1}
W20260924 16:09:22.657702 19458 heartbeater.cc:644] Failed to heartbeat to 127.3.214.254:38473 (0 consecutive failures): Network error: Failed to ping master at 127.3.214.254:38473: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
I20260924 16:09:22.658857 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:22.658469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.194:46719)
0924 16:09:22.658640 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:22.658719 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 16:09:22.659871 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:22.659453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.194:44575)
0924 16:09:22.659609 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:22.659709 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 16:09:22.660175 19458 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:09:23.053243 19461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:23.053608 19461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:23.053715 19461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:23.061620 19461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:23.061771 19461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:09:23.061898 19461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:09:23.073204 19461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:42317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=35891
--tserver_master_addrs=127.3.214.254:38473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:23.074939 19461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:23.076611 19461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:23.092796 19466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:23.092842 19467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:23.092893 19469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:23.094163 19461 server_base.cc:1034] running on GCE node
I20260924 16:09:23.095023 19461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:23.096342 19461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:09:23.097586 19461 hybrid_clock.cc:648] HybridClock initialized: now 1790266163097509 us; error 74 us; skew 500 ppm
I20260924 16:09:23.097980 19461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:23 dist-test-slave-1gc3 krb5kdc[18000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266163, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:23.101790 19461 init.cc:377] Logged in from keytab as kudu/127.3.214.195@KRBTEST.COM (short username kudu)
I20260924 16:09:23.103796 19461 webserver.cc:460] Webserver started at http://127.3.214.195:35891/ using document root <none> and password file <none>
I20260924 16:09:23.104710 19461 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:23.104843 19461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:09:23.111877 19461 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 16:09:23.115736 19476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:23.117674 19461 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:09:23.117894 19461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "065aa5432a694522acaa7a486a720a31"
format_stamp: "Formatted at 2026-09-24 16:09:16 on dist-test-slave-1gc3"
server_key: "13e70b2a461cc85d44c301125c5735b0"
server_key_iv: "3f66d4de4d6fa2817e9cbacab03ae81e"
server_key_version: "encryptionkey@0"
I20260924 16:09:23.119232 19461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:23.139746 19461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:23.142452 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:23.141336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.193:42889)
0924 16:09:23.141984 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:23.142218 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":314,"threads_started":1}
I20260924 16:09:23.145725 19461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:23.146068 19461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:23.147495 19461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:09:23.149257 19461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:09:23.149377 19461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:23.149501 19461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:09:23.149580 19461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:23.188577 19461 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:42317
I20260924 16:09:23.188655 19590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:42317 every 8 connection(s)
I20260924 16:09:23.190455 19461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:09:23.194515 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 19461
I20260924 16:09:23.196906 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:23.194190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.195:54979)
0924 16:09:23.195338 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:23.195625 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":395,"threads_started":1}
W20260924 16:09:23.197741 19591 heartbeater.cc:644] Failed to heartbeat to 127.3.214.254:38473 (0 consecutive failures): Network error: Failed to ping master at 127.3.214.254:38473: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
I20260924 16:09:23.198675 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:23.198327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.195:56431)
0924 16:09:23.198481 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:23.198552 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 16:09:23.199496 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:23.199176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.195:52413)
0924 16:09:23.199314 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:23.199388 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 16:09:23.199733 19591 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 16:09:23.405068 19595 raft_consensus.cc:482] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:09:23.405570 19595 raft_consensus.cc:504] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } } peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } }
I20260924 16:09:23.408020 19595 leader_election.cc:288] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4252c8af4a7745c3b0eef21e54f18827 (127.3.214.194:42017), 065aa5432a694522acaa7a486a720a31 (127.3.214.195:42317)
Sep 24 16:09:23 dist-test-slave-1gc3 krb5kdc[18000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.195@KRBTEST.COM
Sep 24 16:09:23 dist-test-slave-1gc3 krb5kdc[18000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.194@KRBTEST.COM
I20260924 16:09:23.441828 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:23.408537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:42017 (local address 127.3.214.193:48153)
0924 16:09:23.408759 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:23.408828 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:23.409494 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:23.413023 (+ 3529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:23.413036 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:23.414010 (+ 974us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:23.414874 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:23.414899 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:23.416692 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:23.416696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:23.418568 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:23.418594 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:23.419781 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:23.419789 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:23.420070 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:23.426799 (+ 6729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:23.426850 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:23.430457 (+ 3607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:23.435787 (+ 5330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:23.435818 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:23.435833 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:23.436929 (+ 1096us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:23.437726 (+ 797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:23.437733 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:23.437737 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:23.438083 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:23.439362 (+ 1279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:23.439374 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:23.440073 (+ 699us) client_negotiation.cc:769] Sending connection context
0924 16:09:23.440515 (+ 442us) client_negotiation.cc:240] Negotiation successful
0924 16:09:23.441331 (+ 816us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":228,"spinlock_wait_cycles":50816}
I20260924 16:09:23.441910 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:23.408750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:48153 (local address 127.3.214.194:42017)
0924 16:09:23.409889 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:23.409914 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:23.409985 (+ 71us) server_negotiation.cc:407] Connection header received
0924 16:09:23.411910 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:23.411925 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:23.412455 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:23.412807 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:23.415116 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:23.416464 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:23.418753 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:23.419508 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:23.419716 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:23.430911 (+ 11195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:23.430926 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:23.430955 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:23.431020 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:23.435470 (+ 4450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:23.437208 (+ 1738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:23.437220 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:23.437233 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:23.437547 (+ 314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:23.438272 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:23.438278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:23.438281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:23.439150 (+ 869us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:23.439379 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:23.440915 (+ 1536us) server_negotiation.cc:299] Negotiation successful
0924 16:09:23.441497 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":917,"thread_start_us":774,"threads_started":1}
I20260924 16:09:23.442203 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:23.409710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:48841 (local address 127.3.214.195:42317)
0924 16:09:23.410414 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:23.410429 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:23.410837 (+ 408us) server_negotiation.cc:407] Connection header received
0924 16:09:23.412537 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:23.412546 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:23.413036 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:23.413439 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:23.415123 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:23.416374 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:23.418729 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:23.419429 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:23.419585 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:23.430911 (+ 11326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:23.430926 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:23.430954 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:23.431020 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:23.435470 (+ 4450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:23.437208 (+ 1738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:23.437220 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:23.437233 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:23.437547 (+ 314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:23.438272 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:23.438278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:23.438281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:23.439152 (+ 871us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:23.439379 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:23.441199 (+ 1820us) server_negotiation.cc:299] Negotiation successful
0924 16:09:23.441728 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":312,"threads_started":1}
I20260924 16:09:23.443176 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:23.409026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:42317 (local address 127.3.214.193:48841)
0924 16:09:23.410717 (+ 1691us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:23.410733 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:23.410860 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:23.413430 (+ 2570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:23.413435 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:23.414160 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:23.414877 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:23.414900 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:23.416586 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:23.416592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:23.418573 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:23.418589 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:23.419646 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:23.419653 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:23.420070 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:23.426805 (+ 6735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:23.426840 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:23.430320 (+ 3480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:23.435786 (+ 5466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:23.435804 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:23.435833 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:23.436929 (+ 1096us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:23.437726 (+ 797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:23.437732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:23.437736 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:23.438083 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:23.439384 (+ 1301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:23.439388 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:23.440073 (+ 685us) client_negotiation.cc:769] Sending connection context
0924 16:09:23.440515 (+ 442us) client_negotiation.cc:240] Negotiation successful
0924 16:09:23.442861 (+ 2346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1574,"mutex_wait_us":2052,"spinlock_wait_cycles":50688,"thread_start_us":1383,"threads_started":1}
W20260924 16:09:23.447367 19213 leader_election.cc:341] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 065aa5432a694522acaa7a486a720a31 (127.3.214.195:42317): Not found: Tablet not found: 96d4659136634e2bb6a42c4e431efd34
W20260924 16:09:23.447706 19213 leader_election.cc:341] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4252c8af4a7745c3b0eef21e54f18827 (127.3.214.194:42017): Not found: Tablet not found: 96d4659136634e2bb6a42c4e431efd34
I20260924 16:09:23.447816 19213 leader_election.cc:302] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1652cd36cab4e51a07e4d1fb6e361df; no voters: 065aa5432a694522acaa7a486a720a31, 4252c8af4a7745c3b0eef21e54f18827
I20260924 16:09:23.448339 19595 raft_consensus.cc:2726] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:09:23.662423 19599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:23.661083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.194:59255)
0924 16:09:23.661769 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:23.661998 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":336,"threads_started":1}
I20260924 16:09:24.144619 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:24.143682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.193:38279)
0924 16:09:24.144257 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:24.144403 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":279,"threads_started":1}
I20260924 16:09:24.201435 19601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:24.200351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.195:60415)
0924 16:09:24.201041 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:24.201174 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":393,"threads_started":1}
I20260924 16:09:24.664770 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:24.663744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.194:54779)
0924 16:09:24.664437 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:24.664548 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":344,"threads_started":1}
I20260924 16:09:25.146872 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:25.145857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.193:45359)
0924 16:09:25.146521 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:25.146645 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":341,"threads_started":1}
I20260924 16:09:25.201402 19604 raft_consensus.cc:482] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:09:25.201699 19604 raft_consensus.cc:504] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } } peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } }
I20260924 16:09:25.202579 19604 leader_election.cc:288] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4252c8af4a7745c3b0eef21e54f18827 (127.3.214.194:42017), 065aa5432a694522acaa7a486a720a31 (127.3.214.195:42317)
I20260924 16:09:25.203720 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:25.202574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.195:40397)
0924 16:09:25.203263 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:25.203424 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":334,"threads_started":1}
W20260924 16:09:25.204106 19213 leader_election.cc:341] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4252c8af4a7745c3b0eef21e54f18827 (127.3.214.194:42017): Not found: Tablet not found: 96d4659136634e2bb6a42c4e431efd34
W20260924 16:09:25.204368 19213 leader_election.cc:341] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 065aa5432a694522acaa7a486a720a31 (127.3.214.195:42317): Not found: Tablet not found: 96d4659136634e2bb6a42c4e431efd34
I20260924 16:09:25.204458 19213 leader_election.cc:302] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1652cd36cab4e51a07e4d1fb6e361df; no voters: 065aa5432a694522acaa7a486a720a31, 4252c8af4a7745c3b0eef21e54f18827
I20260924 16:09:25.204782 19604 raft_consensus.cc:2726] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:09:25.666865 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:25.665841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.194:43901)
0924 16:09:25.666542 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:25.666655 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":349,"threads_started":1}
I20260924 16:09:26.148998 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:26.148052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.193:41783)
0924 16:09:26.148678 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:26.148792 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":333,"threads_started":1}
I20260924 16:09:26.205870 19608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:26.204863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.195:58037)
0924 16:09:26.205493 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:26.205616 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":323,"threads_started":1}
I20260924 16:09:26.669013 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:26.668015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.194:59573)
0924 16:09:26.668681 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:26.668806 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":346,"threads_started":1}
I20260924 16:09:27.151436 19610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:27.150383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.193:54191)
0924 16:09:27.151029 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:27.151152 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":285,"threads_started":1}
I20260924 16:09:27.155016 19611 raft_consensus.cc:482] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:09:27.155229 19611 raft_consensus.cc:504] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } } peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } }
I20260924 16:09:27.155961 19611 leader_election.cc:288] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4252c8af4a7745c3b0eef21e54f18827 (127.3.214.194:42017), 065aa5432a694522acaa7a486a720a31 (127.3.214.195:42317)
W20260924 16:09:27.157137 19213 leader_election.cc:341] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 065aa5432a694522acaa7a486a720a31 (127.3.214.195:42317): Not found: Tablet not found: 96d4659136634e2bb6a42c4e431efd34
W20260924 16:09:27.157410 19213 leader_election.cc:341] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4252c8af4a7745c3b0eef21e54f18827 (127.3.214.194:42017): Not found: Tablet not found: 96d4659136634e2bb6a42c4e431efd34
I20260924 16:09:27.157536 19213 leader_election.cc:302] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1652cd36cab4e51a07e4d1fb6e361df; no voters: 065aa5432a694522acaa7a486a720a31, 4252c8af4a7745c3b0eef21e54f18827
I20260924 16:09:27.157817 19611 raft_consensus.cc:2726] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:09:27.208185 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:27.207174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.195:53949)
0924 16:09:27.207823 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:27.207957 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":306,"threads_started":1}
I20260924 16:09:27.671114 19613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:27.670285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.194:38227)
0924 16:09:27.670837 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:27.670933 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":292,"threads_started":1}
I20260924 16:09:28.153657 19614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:28.152661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.193:45563)
0924 16:09:28.153291 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:28.153444 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":305,"threads_started":1}
I20260924 16:09:28.210598 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:28.209563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.195:57175)
0924 16:09:28.210244 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:28.210386 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":333,"threads_started":1}
I20260924 16:09:28.673454 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:28.672515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.194:39901)
0924 16:09:28.673150 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:28.673263 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":321,"threads_started":1}
I20260924 16:09:29.156437 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:29.155115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.193:55777)
0924 16:09:29.155967 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:29.156120 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":364,"threads_started":1}
I20260924 16:09:29.213039 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:29.211937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.195:38695)
0924 16:09:29.212660 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:29.212800 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":396,"threads_started":1}
I20260924 16:09:29.631783 19619 raft_consensus.cc:482] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:09:29.632138 19619 raft_consensus.cc:504] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } } peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } }
I20260924 16:09:29.633029 19619 leader_election.cc:288] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4252c8af4a7745c3b0eef21e54f18827 (127.3.214.194:42017), 065aa5432a694522acaa7a486a720a31 (127.3.214.195:42317)
W20260924 16:09:29.634285 19213 leader_election.cc:341] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 065aa5432a694522acaa7a486a720a31 (127.3.214.195:42317): Not found: Tablet not found: 96d4659136634e2bb6a42c4e431efd34
W20260924 16:09:29.634579 19213 leader_election.cc:341] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4252c8af4a7745c3b0eef21e54f18827 (127.3.214.194:42017): Not found: Tablet not found: 96d4659136634e2bb6a42c4e431efd34
I20260924 16:09:29.634662 19213 leader_election.cc:302] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1652cd36cab4e51a07e4d1fb6e361df; no voters: 065aa5432a694522acaa7a486a720a31, 4252c8af4a7745c3b0eef21e54f18827
I20260924 16:09:29.634966 19619 raft_consensus.cc:2726] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:09:29.675817 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:29.674763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.194:34145)
0924 16:09:29.675488 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:29.675607 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":310,"threads_started":1}
I20260924 16:09:30.158780 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:30.157789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.193:38641)
0924 16:09:30.158429 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:30.158567 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":329,"threads_started":1}
I20260924 16:09:30.215557 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:30.214388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.195:46843)
0924 16:09:30.215128 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:30.215248 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":319,"threads_started":1}
I20260924 16:09:30.678094 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:30.676997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.194:54971)
0924 16:09:30.677714 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:30.677847 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":379,"threads_started":1}
I20260924 16:09:31.161201 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:31.160081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.193:44617)
0924 16:09:31.160844 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:31.160976 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":329,"threads_started":1}
I20260924 16:09:31.217732 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:31.216719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.195:38881)
0924 16:09:31.217380 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:31.217505 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":314,"threads_started":1}
I20260924 16:09:31.680331 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:31.679383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.194:52973)
0924 16:09:31.679979 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:31.680148 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":296,"threads_started":1}
I20260924 16:09:32.163652 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:32.162593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.193:56835)
0924 16:09:32.163221 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:32.163377 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":348,"threads_started":1}
I20260924 16:09:32.219861 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:32.218931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.195:50265)
0924 16:09:32.219579 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:32.219682 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":313,"threads_started":1}
I20260924 16:09:32.682614 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:32.681630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.194:40895)
0924 16:09:32.682313 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:32.682418 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":441,"threads_started":1}
I20260924 16:09:33.165848 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:33.164801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.193:41679)
0924 16:09:33.165487 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:33.165621 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":311,"threads_started":1}
I20260924 16:09:33.222146 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:33.221104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.195:59039)
0924 16:09:33.221747 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:33.221873 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":337,"threads_started":1}
I20260924 16:09:33.684787 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:33.683695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.194:42571)
0924 16:09:33.684423 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:33.684561 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":332,"threads_started":1}
I20260924 16:09:34.168105 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:34.167190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.193:39591)
0924 16:09:34.167792 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:34.167899 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":315,"threads_started":1}
I20260924 16:09:34.224577 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:34.223493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.195:57155)
0924 16:09:34.224268 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:34.224387 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":411,"threads_started":1}
I20260924 16:09:34.687217 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:34.686200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.194:53313)
0924 16:09:34.686872 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:34.686992 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":364,"threads_started":1}
I20260924 16:09:35.170455 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:35.169352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.193:50865)
0924 16:09:35.170027 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:35.170203 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":349,"threads_started":1}
I20260924 16:09:35.226768 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:35.225785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.195:43763)
0924 16:09:35.226442 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:35.226555 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":333,"threads_started":1}
I20260924 16:09:35.689432 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:35.688426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.194:45779)
0924 16:09:35.689030 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:35.689162 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":311,"threads_started":1}
I20260924 16:09:36.172659 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:36.171731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.193:59745)
0924 16:09:36.172338 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:36.172452 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":299,"threads_started":1}
I20260924 16:09:36.228982 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:36.227984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.195:41605)
0924 16:09:36.228641 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:36.228790 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":342,"threads_started":1}
I20260924 16:09:36.691879 19641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:36.690829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.194:49859)
0924 16:09:36.691531 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:36.691656 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":350,"threads_started":1}
I20260924 16:09:36.776515 19642 raft_consensus.cc:482] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:09:36.776767 19642 raft_consensus.cc:504] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4252c8af4a7745c3b0eef21e54f18827" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42017 } } peers { permanent_uuid: "f1652cd36cab4e51a07e4d1fb6e361df" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34359 } } peers { permanent_uuid: "065aa5432a694522acaa7a486a720a31" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42317 } }
I20260924 16:09:36.777539 19642 leader_election.cc:288] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4252c8af4a7745c3b0eef21e54f18827 (127.3.214.194:42017), 065aa5432a694522acaa7a486a720a31 (127.3.214.195:42317)
W20260924 16:09:36.778703 19213 leader_election.cc:341] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 065aa5432a694522acaa7a486a720a31 (127.3.214.195:42317): Not found: Tablet not found: 96d4659136634e2bb6a42c4e431efd34
W20260924 16:09:36.778909 19213 leader_election.cc:341] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4252c8af4a7745c3b0eef21e54f18827 (127.3.214.194:42017): Not found: Tablet not found: 96d4659136634e2bb6a42c4e431efd34
I20260924 16:09:36.779021 19213 leader_election.cc:302] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1652cd36cab4e51a07e4d1fb6e361df; no voters: 065aa5432a694522acaa7a486a720a31, 4252c8af4a7745c3b0eef21e54f18827
I20260924 16:09:36.779356 19642 raft_consensus.cc:2726] T 96d4659136634e2bb6a42c4e431efd34 P f1652cd36cab4e51a07e4d1fb6e361df [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:09:37.175208 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:37.174222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.193:35583)
0924 16:09:37.174836 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:37.174986 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":303,"threads_started":1}
I20260924 16:09:37.231112 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:37.230172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.195:43693)
0924 16:09:37.230794 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:37.230909 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":334,"threads_started":1}
I20260924 16:09:37.694342 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:37.693368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.194:42859)
0924 16:09:37.693957 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:37.694149 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":307,"threads_started":1}
I20260924 16:09:38.177562 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:38.176361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.193:42121)
0924 16:09:38.177098 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:38.177232 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":360,"threads_started":1}
I20260924 16:09:38.201741 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 19160
I20260924 16:09:38.227799 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:38.203947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56962 (local address 127.3.214.193:34359)
0924 16:09:38.204831 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:38.204845 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:38.204900 (+ 55us) server_negotiation.cc:407] Connection header received
0924 16:09:38.205016 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:38.205028 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:38.205423 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:38.205650 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:38.207326 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:38.208528 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:38.209944 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:38.210754 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:38.210914 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:38.220216 (+ 9302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:38.220259 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:38.220288 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:38.220360 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:38.224162 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:38.225281 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:38.225292 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:38.225295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:38.225477 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:38.225870 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:38.225874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:38.225877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:38.226417 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:38.226742 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:38.227071 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 16:09:38.227344 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":315,"threads_started":1}
I20260924 16:09:38.229470 19322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:38.234745 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:38.232776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:38473 (local address 127.3.214.195:34381)
0924 16:09:38.234389 (+ 1613us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:38.234532 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:38473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1448,"thread_start_us":1212,"threads_started":1}
I20260924 16:09:38.324487 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 19160
I20260924 16:09:38.344017 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 19330
I20260924 16:09:38.360798 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:38.344914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34816 (local address 127.3.214.194:42017)
0924 16:09:38.345610 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:38.345616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:38.345640 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:09:38.345721 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:38.345726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:38.345807 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:38.345926 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:38.347866 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:38.348866 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:38.350503 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:38.351015 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:38.351145 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:38.354283 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:38.354292 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:38.354297 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:38.354393 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:38.357551 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:38.358626 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:38.358631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:38.358634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:38.358819 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:38.359195 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:38.359198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:38.359200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:38.359542 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:38.359808 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:38.360105 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 16:09:38.360378 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":350,"threads_started":1}
I20260924 16:09:38.361796 19453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:38.452119 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 19330
I20260924 16:09:38.472699 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 19461
I20260924 16:09:38.488807 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:38.473873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33748 (local address 127.3.214.195:42317)
0924 16:09:38.474595 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:38.474600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:38.474617 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:09:38.474703 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:38.474707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:38.474780 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:38.474877 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:38.476737 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:38.477603 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:38.479223 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:38.479740 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:38.479874 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:38.482806 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:38.482812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:38.482816 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:38.482865 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:38.485654 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:38.486586 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:38.486590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:38.486592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:38.486764 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:38.487192 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:38.487196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:38.487198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:38.487507 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:38.487781 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:38.488140 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 16:09:38.488371 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":392,"threads_started":1}
I20260924 16:09:38.489842 19586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:38.583520 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 19461
2026-09-24T16:09:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25597 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:09:38 dist-test-slave-1gc3 krb5kdc[19656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:09:38 dist-test-slave-1gc3 krb5kdc[19656](info): set up 2 sockets
Sep 24 16:09:38 dist-test-slave-1gc3 krb5kdc[19656](info): commencing operation
krb5kdc: starting...
W20260924 16:09:39.656215 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:09:39 dist-test-slave-1gc3 krb5kdc[19656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:09:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:09:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:09:39.960484 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:40829
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:40829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:09:40.371199 19669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:40.371618 19669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:40.371754 19669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:40.380051 19669 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 16:09:40.380187 19669 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 16:09:40.380227 19669 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 16:09:40.380288 19669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:09:40.380370 19669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:40.380465 19669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:09:40.380546 19669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:09:40.380625 19669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:09:40.392131 19669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:40829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:40829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:40.393939 19669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:40.395730 19669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:40.410431 19675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:40.410539 19674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:40.410539 19677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:40.411176 19669 server_base.cc:1034] running on GCE node
I20260924 16:09:40.412250 19669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:40.413841 19669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:09:40.415119 19669 hybrid_clock.cc:648] HybridClock initialized: now 1790266180415000 us; error 80 us; skew 500 ppm
I20260924 16:09:40.415513 19669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:40 dist-test-slave-1gc3 krb5kdc[19656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266180, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:40.419006 19669 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:09:40.420675 19669 webserver.cc:460] Webserver started at http://127.3.214.254:38721/ using document root <none> and password file <none>
I20260924 16:09:40.421469 19669 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:40.421576 19669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:09:40.421901 19669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:09:40.425247 19669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "2ae01576d2fc4609972bb444423d12a7"
format_stamp: "Formatted at 2026-09-24 16:09:40 on dist-test-slave-1gc3"
server_key: "75d417bd5c0bf24aac1c7a7c34763393"
server_key_iv: "72257ba305e7e54adba06fc624aad30e"
server_key_version: "encryptionkey@0"
I20260924 16:09:40.426048 19669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "2ae01576d2fc4609972bb444423d12a7"
format_stamp: "Formatted at 2026-09-24 16:09:40 on dist-test-slave-1gc3"
server_key: "75d417bd5c0bf24aac1c7a7c34763393"
server_key_iv: "72257ba305e7e54adba06fc624aad30e"
server_key_version: "encryptionkey@0"
I20260924 16:09:40.432085 19669 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260924 16:09:40.436352 19684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:40.438203 19669 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:09:40.438406 19669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "2ae01576d2fc4609972bb444423d12a7"
format_stamp: "Formatted at 2026-09-24 16:09:40 on dist-test-slave-1gc3"
server_key: "75d417bd5c0bf24aac1c7a7c34763393"
server_key_iv: "72257ba305e7e54adba06fc624aad30e"
server_key_version: "encryptionkey@0"
I20260924 16:09:40.438640 19669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:40.462224 19669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:40.466871 19669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:40.467124 19669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:40.488225 19669 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:40829
I20260924 16:09:40.488258 19736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:40829 every 8 connection(s)
I20260924 16:09:40.489725 19669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:09:40.493844 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 19669
I20260924 16:09:40.494071 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:09:40.494786 3931 external_mini_cluster.cc:1428] Setting key 5ffe3d977621d860863650561e5c19b9
I20260924 16:09:40.495014 19737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:09:40.510915 19737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7: Bootstrap starting.
Sep 24 16:09:40 dist-test-slave-1gc3 krb5kdc[19656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:09:40.515388 19737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7: Neither blocks nor log segments found. Creating new log.
I20260924 16:09:40.516810 19737 log.cc:824] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7: Log is configured to *not* fsync() on all Append() calls
I20260924 16:09:40.521150 19737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7: No bootstrap required, opened a new log
I20260924 16:09:40.521849 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:40.497140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43962 (local address 127.3.214.254:40829)
0924 16:09:40.498212 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:40.498229 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:40.498281 (+ 52us) server_negotiation.cc:407] Connection header received
0924 16:09:40.500900 (+ 2619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:40.500933 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:40.501469 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:40.501921 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:40.503703 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:40.505283 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:40.506805 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:40.507637 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:40.507847 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:40.512549 (+ 4702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:40.512559 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:40.512580 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:40.512635 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:40.516202 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:40.516997 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:40.517007 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:40.517017 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:40.517221 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:40.517730 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:40.517735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:40.517738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:40.518712 (+ 974us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:40.519125 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:40.519609 (+ 484us) server_negotiation.cc:299] Negotiation successful
0924 16:09:40.520200 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":429,"threads_started":1}
I20260924 16:09:40.530184 19737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae01576d2fc4609972bb444423d12a7" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40829 } }
I20260924 16:09:40.530665 19737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:09:40.530797 19737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae01576d2fc4609972bb444423d12a7, State: Initialized, Role: FOLLOWER
I20260924 16:09:40.531500 19737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae01576d2fc4609972bb444423d12a7" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40829 } }
I20260924 16:09:40.531755 19737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:09:40.531900 19737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:09:40.532042 19737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:09:40.534650 19737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae01576d2fc4609972bb444423d12a7" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40829 } }
I20260924 16:09:40.535384 19737 leader_election.cc:302] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ae01576d2fc4609972bb444423d12a7; no voters:
I20260924 16:09:40.536098 19737 leader_election.cc:288] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:09:40.536303 19742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:09:40.537155 19742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 1 LEADER]: Becoming Leader. State: Replica: 2ae01576d2fc4609972bb444423d12a7, State: Running, Role: LEADER
I20260924 16:09:40.537870 19742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae01576d2fc4609972bb444423d12a7" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40829 } }
I20260924 16:09:40.538586 19737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:09:40.541402 19744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ae01576d2fc4609972bb444423d12a7. Latest consensus state: current_term: 1 leader_uuid: "2ae01576d2fc4609972bb444423d12a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae01576d2fc4609972bb444423d12a7" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40829 } } }
I20260924 16:09:40.541952 19744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:09:40.541400 19743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ae01576d2fc4609972bb444423d12a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae01576d2fc4609972bb444423d12a7" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40829 } } }
I20260924 16:09:40.544426 19743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:09:40.546063 19750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:09:40.553105 19750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:09:40.566216 19750 catalog_manager.cc:1347] Generated new cluster ID: e809d8a952f6463ab15ca057e2e792a5
I20260924 16:09:40.566418 19750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:09:40.581703 19750 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:09:40.583213 19750 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:09:40.590796 19750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7: Generated new TSK 0
I20260924 16:09:40.591984 19750 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.214.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
I20260924 16:09:40.682574 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:40829
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:09:41.127861 19763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:41.128268 19763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:41.128423 19763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:41.136698 19763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:41.136858 19763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:09:41.137039 19763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:09:41.148552 19763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:40829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:41.150498 19763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:41.152215 19763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:41.167662 19768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:41.168147 19771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:41.168066 19769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:41.169281 19763 server_base.cc:1034] running on GCE node
I20260924 16:09:41.170249 19763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:41.171743 19763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:09:41.173015 19763 hybrid_clock.cc:648] HybridClock initialized: now 1790266181172903 us; error 98 us; skew 500 ppm
I20260924 16:09:41.173435 19763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:41 dist-test-slave-1gc3 krb5kdc[19656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266181, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:41.177496 19763 init.cc:377] Logged in from keytab as kudu/127.3.214.193@KRBTEST.COM (short username kudu)
I20260924 16:09:41.179585 19763 webserver.cc:460] Webserver started at http://127.3.214.193:43921/ using document root <none> and password file <none>
I20260924 16:09:41.180495 19763 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:41.180632 19763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:09:41.181010 19763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:09:41.184616 19763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "32aa91a44d354548a6a62b87606e372f"
format_stamp: "Formatted at 2026-09-24 16:09:41 on dist-test-slave-1gc3"
server_key: "9c78082edc6a91d8166f6c3e36eceab8"
server_key_iv: "cd71c2897678fbb9056137e0c837911e"
server_key_version: "encryptionkey@0"
I20260924 16:09:41.185513 19763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "32aa91a44d354548a6a62b87606e372f"
format_stamp: "Formatted at 2026-09-24 16:09:41 on dist-test-slave-1gc3"
server_key: "9c78082edc6a91d8166f6c3e36eceab8"
server_key_iv: "cd71c2897678fbb9056137e0c837911e"
server_key_version: "encryptionkey@0"
I20260924 16:09:41.191993 19763 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260924 16:09:41.196373 19778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:41.198091 19763 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260924 16:09:41.198307 19763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "32aa91a44d354548a6a62b87606e372f"
format_stamp: "Formatted at 2026-09-24 16:09:41 on dist-test-slave-1gc3"
server_key: "9c78082edc6a91d8166f6c3e36eceab8"
server_key_iv: "cd71c2897678fbb9056137e0c837911e"
server_key_version: "encryptionkey@0"
I20260924 16:09:41.198550 19763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:41.224948 19763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:41.230598 19763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:41.230973 19763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:41.232394 19763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:09:41.234279 19763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:09:41.234376 19763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:41.234539 19763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:09:41.234607 19763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:41.272959 19763 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:39785
I20260924 16:09:41.273010 19890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:39785 every 8 connection(s)
I20260924 16:09:41.274623 19763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:09:41.283022 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 19763
I20260924 16:09:41.283236 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:09:41.283900 3931 external_mini_cluster.cc:1428] Setting key b6522204f640bbf23c4546141cc6c092
Sep 24 16:09:41 dist-test-slave-1gc3 krb5kdc[19656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266181, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:09:41.298628 19893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:41.277582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:57021 (local address 127.3.214.254:40829)
0924 16:09:41.278229 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:41.278237 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:41.279380 (+ 1143us) server_negotiation.cc:407] Connection header received
0924 16:09:41.281673 (+ 2293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:41.281679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:41.281771 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:41.281915 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:41.284041 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:41.284994 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:41.286977 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:41.287539 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:41.287701 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:41.291767 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:41.291775 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:41.291780 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:41.291831 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:41.295015 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:41.295921 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:41.295926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:41.295930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:41.296104 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:41.296592 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:41.296596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:41.296599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:41.296967 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:41.297125 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:41.298152 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0924 16:09:41.298346 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":279,"threads_started":1}
I20260924 16:09:41.300596 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:41.278134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.193:57021)
0924 16:09:41.279144 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:41.279209 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:41.281291 (+ 2082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:41.282240 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:41.282251 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:41.282975 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:41.283793 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:41.283816 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:41.285206 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:41.285213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:41.286803 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:41.286819 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:41.287790 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:41.287796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:41.288057 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:41.288819 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:41.288846 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:41.291423 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:41.295216 (+ 3793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:41.295225 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:41.295244 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:41.295754 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:41.296227 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:41.296230 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:41.296232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:41.296465 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:41.297105 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:41.297113 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:41.297661 (+ 548us) client_negotiation.cc:769] Sending connection context
0924 16:09:41.298011 (+ 350us) client_negotiation.cc:240] Negotiation successful
0924 16:09:41.298731 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":357,"threads_started":1}
I20260924 16:09:41.302678 19891 heartbeater.cc:347] Connected to a master server at 127.3.214.254:40829
I20260924 16:09:41.303115 19891 heartbeater.cc:464] Registering TS with master...
I20260924 16:09:41.304322 19891 heartbeater.cc:511] Master 127.3.214.254:40829 requested a full tablet report, sending...
I20260924 16:09:41.307312 19701 ts_manager.cc:194] Registered new tserver with Master: 32aa91a44d354548a6a62b87606e372f (127.3.214.193:39785)
I20260924 16:09:41.308938 19701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:57021
WARNING: no policy specified for kudu/127.3.214.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
I20260924 16:09:41.353143 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:40829
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:09:41.758746 19897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:41.759147 19897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:41.759294 19897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:41.767539 19897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:41.767709 19897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:09:41.767901 19897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:09:41.779559 19897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:40829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:41.781356 19897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:41.783130 19897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:41.799230 19902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:41.799193 19903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:41.799372 19905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:41.800386 19897 server_base.cc:1034] running on GCE node
I20260924 16:09:41.801185 19897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:41.802393 19897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:09:41.803671 19897 hybrid_clock.cc:648] HybridClock initialized: now 1790266181803552 us; error 104 us; skew 500 ppm
I20260924 16:09:41.804160 19897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:41 dist-test-slave-1gc3 krb5kdc[19656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266181, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:41.808168 19897 init.cc:377] Logged in from keytab as kudu/127.3.214.194@KRBTEST.COM (short username kudu)
I20260924 16:09:41.810206 19897 webserver.cc:460] Webserver started at http://127.3.214.194:45585/ using document root <none> and password file <none>
I20260924 16:09:41.811045 19897 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:41.811177 19897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:09:41.811515 19897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:09:41.814914 19897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "4d2b7a5863b74f23b9c03656247e4c3a"
format_stamp: "Formatted at 2026-09-24 16:09:41 on dist-test-slave-1gc3"
server_key: "efdacfceba9f987c930af1dbdfc25b85"
server_key_iv: "cecbc330ff8ac6164e4a2812d8ffda9a"
server_key_version: "encryptionkey@0"
I20260924 16:09:41.815784 19897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "4d2b7a5863b74f23b9c03656247e4c3a"
format_stamp: "Formatted at 2026-09-24 16:09:41 on dist-test-slave-1gc3"
server_key: "efdacfceba9f987c930af1dbdfc25b85"
server_key_iv: "cecbc330ff8ac6164e4a2812d8ffda9a"
server_key_version: "encryptionkey@0"
I20260924 16:09:41.822005 19897 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 16:09:41.827399 19912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:41.829536 19897 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:09:41.829785 19897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "4d2b7a5863b74f23b9c03656247e4c3a"
format_stamp: "Formatted at 2026-09-24 16:09:41 on dist-test-slave-1gc3"
server_key: "efdacfceba9f987c930af1dbdfc25b85"
server_key_iv: "cecbc330ff8ac6164e4a2812d8ffda9a"
server_key_version: "encryptionkey@0"
I20260924 16:09:41.830027 19897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:41.867028 19897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:41.872000 19897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:41.872323 19897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:41.873579 19897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:09:41.875618 19897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:09:41.875723 19897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:41.875860 19897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:09:41.875926 19897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:41.916124 19897 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:40967
I20260924 16:09:41.916199 20025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:40967 every 8 connection(s)
I20260924 16:09:41.917832 19897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:09:41.926084 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 19897
I20260924 16:09:41.926327 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:09:41.926940 3931 external_mini_cluster.cc:1428] Setting key c5f0e5e490b5b256b920dbf1f5e871af
Sep 24 16:09:41 dist-test-slave-1gc3 krb5kdc[19656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266181, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:09:41.944260 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:41.920944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:57535 (local address 127.3.214.254:40829)
0924 16:09:41.921563 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:41.921572 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:41.922974 (+ 1402us) server_negotiation.cc:407] Connection header received
0924 16:09:41.925027 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:41.925032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:41.925130 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:41.925271 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:41.927887 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:41.928917 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:41.931289 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:41.931877 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:41.932012 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:41.937065 (+ 5053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:41.937072 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:41.937075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:41.937124 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:41.940168 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:41.941198 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:41.941204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:41.941209 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:41.941364 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:41.941892 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:41.941897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:41.941901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:41.942320 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:41.942486 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:41.943483 (+ 997us) server_negotiation.cc:299] Negotiation successful
0924 16:09:41.943850 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":267,"threads_started":1}
I20260924 16:09:41.945591 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:41.921764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.194:57535)
0924 16:09:41.922751 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:41.922809 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:41.924702 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:41.925601 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:41.925615 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:41.926604 (+ 989us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:41.927568 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:41.927607 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:41.929144 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:41.929148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:41.931034 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:41.931052 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:41.932181 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:41.932189 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:41.932499 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:41.933385 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:41.933417 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:41.936669 (+ 3252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:41.940395 (+ 3726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:41.940407 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:41.940429 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:41.941041 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:41.941487 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:41.941492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:41.941495 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:41.941762 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:41.942458 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:41.942469 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:41.943016 (+ 547us) client_negotiation.cc:769] Sending connection context
0924 16:09:41.943344 (+ 328us) client_negotiation.cc:240] Negotiation successful
0924 16:09:41.944087 (+ 743us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":363,"threads_started":1}
I20260924 16:09:41.948014 20026 heartbeater.cc:347] Connected to a master server at 127.3.214.254:40829
I20260924 16:09:41.948575 20026 heartbeater.cc:464] Registering TS with master...
I20260924 16:09:41.949677 20026 heartbeater.cc:511] Master 127.3.214.254:40829 requested a full tablet report, sending...
I20260924 16:09:41.951843 19701 ts_manager.cc:194] Registered new tserver with Master: 4d2b7a5863b74f23b9c03656247e4c3a (127.3.214.194:40967)
I20260924 16:09:41.952888 19701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:57535
WARNING: no policy specified for kudu/127.3.214.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
I20260924 16:09:41.995010 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:40829
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:09:42.313860 19891 heartbeater.cc:503] Master 127.3.214.254:40829 was elected leader, sending a full tablet report...
W20260924 16:09:42.376143 20032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:09:42.376484 20032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:09:42.376624 20032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:09:42.384922 20032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:09:42.385160 20032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:09:42.385319 20032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:09:42.395915 20032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:40829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:09:42.397365 20032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:09:42.398893 20032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:09:42.416256 20037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:42.416965 20038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:09:42.418344 20040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:09:42.418910 20032 server_base.cc:1034] running on GCE node
I20260924 16:09:42.419763 20032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:09:42.420971 20032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:09:42.422490 20032 hybrid_clock.cc:648] HybridClock initialized: now 1790266182422404 us; error 60 us; skew 500 ppm
I20260924 16:09:42.423215 20032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:09:42 dist-test-slave-1gc3 krb5kdc[19656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266182, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:09:42.427345 20032 init.cc:377] Logged in from keytab as kudu/127.3.214.195@KRBTEST.COM (short username kudu)
I20260924 16:09:42.429255 20032 webserver.cc:460] Webserver started at http://127.3.214.195:41353/ using document root <none> and password file <none>
I20260924 16:09:42.430022 20032 fs_manager.cc:359] Metadata directory not provided
I20260924 16:09:42.430186 20032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:09:42.430555 20032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:09:42.434020 20032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "a0404a3007e14b969155f38cdde5da50"
format_stamp: "Formatted at 2026-09-24 16:09:42 on dist-test-slave-1gc3"
server_key: "ed460bb15d3791f3fed5f64e0c007bb2"
server_key_iv: "223e26642e659b0bf4f3473ad35d25dc"
server_key_version: "encryptionkey@0"
I20260924 16:09:42.435209 20032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "a0404a3007e14b969155f38cdde5da50"
format_stamp: "Formatted at 2026-09-24 16:09:42 on dist-test-slave-1gc3"
server_key: "ed460bb15d3791f3fed5f64e0c007bb2"
server_key_iv: "223e26642e659b0bf4f3473ad35d25dc"
server_key_version: "encryptionkey@0"
I20260924 16:09:42.442207 20032 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 16:09:42.447280 20047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:42.449430 20032 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260924 16:09:42.449667 20032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "a0404a3007e14b969155f38cdde5da50"
format_stamp: "Formatted at 2026-09-24 16:09:42 on dist-test-slave-1gc3"
server_key: "ed460bb15d3791f3fed5f64e0c007bb2"
server_key_iv: "223e26642e659b0bf4f3473ad35d25dc"
server_key_version: "encryptionkey@0"
I20260924 16:09:42.449935 20032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:09:42.468619 20032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:09:42.473336 20032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:09:42.473639 20032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:09:42.474789 20032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:09:42.476276 20032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:09:42.476356 20032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:42.476490 20032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:09:42.476552 20032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:09:42.518455 20032 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:42739
I20260924 16:09:42.518492 20160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:42739 every 8 connection(s)
I20260924 16:09:42.520238 20032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:09:42.528263 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 20032
I20260924 16:09:42.528472 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:09:42.529006 3931 external_mini_cluster.cc:1428] Setting key c76c219b771dbbd9d4ffdc64262a5198
Sep 24 16:09:42 dist-test-slave-1gc3 krb5kdc[19656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266182, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:09:42.543695 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:42.523504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:60369 (local address 127.3.214.254:40829)
0924 16:09:42.523996 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:42.524001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:42.525456 (+ 1455us) server_negotiation.cc:407] Connection header received
0924 16:09:42.527706 (+ 2250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:42.527712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:42.527811 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:42.527915 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:42.529802 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.530798 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.532520 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.532940 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.533058 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:42.537000 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:42.537005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:42.537008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:42.537047 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:42.540255 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:42.541084 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:42.541090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:42.541093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:42.541240 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:42.541690 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:42.541694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:42.541696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:42.542036 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:42.542229 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:42.542997 (+ 768us) server_negotiation.cc:299] Negotiation successful
0924 16:09:42.543271 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":264,"threads_started":1}
I20260924 16:09:42.544896 20164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:42.524268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.195:60369)
0924 16:09:42.525247 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:42.525311 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:42.527366 (+ 2055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:42.528218 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:42.528227 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:42.528906 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:09:42.529604 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:42.529625 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.530982 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.530987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:42.532344 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:42.532359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.533210 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.533219 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:42.533465 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:42.534208 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:09:42.534234 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:09:42.536748 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:09:42.540439 (+ 3691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:42.540448 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:42.540466 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:42.540949 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:42.541355 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:09:42.541359 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:09:42.541363 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:09:42.541575 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:09:42.542175 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:09:42.542185 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:09:42.542596 (+ 411us) client_negotiation.cc:769] Sending connection context
0924 16:09:42.542866 (+ 270us) client_negotiation.cc:240] Negotiation successful
0924 16:09:42.543528 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":360,"threads_started":1}
I20260924 16:09:42.546939 20161 heartbeater.cc:347] Connected to a master server at 127.3.214.254:40829
I20260924 16:09:42.547353 20161 heartbeater.cc:464] Registering TS with master...
I20260924 16:09:42.548199 20161 heartbeater.cc:511] Master 127.3.214.254:40829 requested a full tablet report, sending...
I20260924 16:09:42.550310 19700 ts_manager.cc:194] Registered new tserver with Master: a0404a3007e14b969155f38cdde5da50 (127.3.214.195:42739)
I20260924 16:09:42.551452 19700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:60369
I20260924 16:09:42.560482 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:09:42.581974 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:42.566737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43972 (local address 127.3.214.254:40829)
0924 16:09:42.567174 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:42.567183 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:42.567300 (+ 117us) server_negotiation.cc:407] Connection header received
0924 16:09:42.567513 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:42.567517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:42.567599 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:42.567715 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:42.569653 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.570542 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.572359 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.572795 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.572913 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:42.575994 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:42.575999 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:42.576003 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:42.576043 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:42.578792 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:42.579807 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:42.579811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:42.579813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:42.579947 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:42.580450 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:42.580454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:42.580456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:42.580847 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:42.581064 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:42.581512 (+ 448us) server_negotiation.cc:299] Negotiation successful
0924 16:09:42.581726 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260924 16:09:42.587953 19700 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:09:42.597175 19700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:09:42.628049 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:42.614723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:39785 (local address 127.0.0.1:59778)
0924 16:09:42.616219 (+ 1496us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:42.616241 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:42.616367 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:42.618792 (+ 2425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:42.618802 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:42.618845 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:09:42.619781 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:42.619798 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.622230 (+ 2432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.622236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:42.624711 (+ 2475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:42.624728 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.627000 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.627006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:42.627207 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:42.627239 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 16:09:42.627341 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 16:09:42.627573 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1350,"mutex_wait_us":137,"thread_start_us":2354,"threads_started":1}
I20260924 16:09:42.628065 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:42.615324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59778 (local address 127.3.214.193:39785)
0924 16:09:42.617868 (+ 2544us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:42.617880 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:42.617917 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:09:42.618062 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:42.618072 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:42.618479 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:42.618721 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:09:42.620105 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.621795 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.624981 (+ 3186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.626762 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.626920 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:42.627226 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:42.627425 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 16:09:42.627658 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2324,"thread_start_us":2142,"threads_started":1}
I20260924 16:09:42.628476 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:42.616128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46082 (local address 127.3.214.194:40967)
0924 16:09:42.618205 (+ 2077us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:42.618213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:42.618244 (+ 31us) server_negotiation.cc:407] Connection header received
0924 16:09:42.618345 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:42.618352 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:42.618590 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:42.619339 (+ 749us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:09:42.620406 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.621543 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.625126 (+ 3583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.626273 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.626435 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:42.626620 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:42.627935 (+ 1315us) server_negotiation.cc:299] Negotiation successful
0924 16:09:42.628158 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1864,"thread_start_us":1747,"threads_started":1}
I20260924 16:09:42.628863 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:42.615808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:40967 (local address 127.0.0.1:46082)
0924 16:09:42.616855 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:42.616875 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:42.616991 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:42.618836 (+ 1845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:42.618840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:42.618860 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:09:42.620205 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:42.620219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.621696 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.621700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:42.624852 (+ 3152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:42.624880 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.626761 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.626766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:42.626965 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:42.627062 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 16:09:42.627214 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 16:09:42.627484 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"mutex_wait_us":157,"thread_start_us":799,"threads_started":1}
I20260924 16:09:42.629284 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:42.614143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:42739 (local address 127.0.0.1:52214)
0924 16:09:42.615278 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:42.615332 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:42.615565 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:42.618934 (+ 3369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:42.618938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:42.618962 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:09:42.619619 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:42.619634 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.621942 (+ 2308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.621948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:42.624852 (+ 2904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:42.624880 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.627465 (+ 2585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.627472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:42.627704 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:42.627741 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 16:09:42.627845 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 16:09:42.628028 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":226,"thread_start_us":616,"threads_started":1}
I20260924 16:09:42.630596 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:42.614387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52214 (local address 127.3.214.195:42739)
0924 16:09:42.617930 (+ 3543us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:42.617938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:42.617969 (+ 31us) server_negotiation.cc:407] Connection header received
0924 16:09:42.618063 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:42.618073 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:42.618479 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:42.618721 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:09:42.619875 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.621795 (+ 1920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.625090 (+ 3295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.626762 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.626929 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:42.627219 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:42.629935 (+ 2716us) server_negotiation.cc:299] Negotiation successful
0924 16:09:42.630200 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3356,"thread_start_us":3165,"threads_started":1}
I20260924 16:09:42.633169 19961 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba071c81ec946d29ffe103da44f9ba5 (DEFAULT_TABLE table=test-table [id=43282d023bf34113ab57072968775a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:09:42.633421 19826 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba071c81ec946d29ffe103da44f9ba5 (DEFAULT_TABLE table=test-table [id=43282d023bf34113ab57072968775a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:09:42.635380 19826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba071c81ec946d29ffe103da44f9ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:09:42.634752 20095 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba071c81ec946d29ffe103da44f9ba5 (DEFAULT_TABLE table=test-table [id=43282d023bf34113ab57072968775a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:09:42.635558 19961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba071c81ec946d29ffe103da44f9ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:09:42.636312 20095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba071c81ec946d29ffe103da44f9ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:09:42.650908 20180 tablet_bootstrap.cc:492] T 6ba071c81ec946d29ffe103da44f9ba5 P 32aa91a44d354548a6a62b87606e372f: Bootstrap starting.
I20260924 16:09:42.655232 20182 tablet_bootstrap.cc:492] T 6ba071c81ec946d29ffe103da44f9ba5 P a0404a3007e14b969155f38cdde5da50: Bootstrap starting.
I20260924 16:09:42.655671 20180 tablet_bootstrap.cc:654] T 6ba071c81ec946d29ffe103da44f9ba5 P 32aa91a44d354548a6a62b87606e372f: Neither blocks nor log segments found. Creating new log.
I20260924 16:09:42.656469 20181 tablet_bootstrap.cc:492] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a: Bootstrap starting.
I20260924 16:09:42.657451 20180 log.cc:824] T 6ba071c81ec946d29ffe103da44f9ba5 P 32aa91a44d354548a6a62b87606e372f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:09:42.661281 20181 tablet_bootstrap.cc:654] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a: Neither blocks nor log segments found. Creating new log.
I20260924 16:09:42.661495 20182 tablet_bootstrap.cc:654] T 6ba071c81ec946d29ffe103da44f9ba5 P a0404a3007e14b969155f38cdde5da50: Neither blocks nor log segments found. Creating new log.
I20260924 16:09:42.662953 20181 log.cc:824] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a: Log is configured to *not* fsync() on all Append() calls
I20260924 16:09:42.663049 20182 log.cc:824] T 6ba071c81ec946d29ffe103da44f9ba5 P a0404a3007e14b969155f38cdde5da50: Log is configured to *not* fsync() on all Append() calls
I20260924 16:09:42.665771 20180 tablet_bootstrap.cc:492] T 6ba071c81ec946d29ffe103da44f9ba5 P 32aa91a44d354548a6a62b87606e372f: No bootstrap required, opened a new log
I20260924 16:09:42.665882 20182 tablet_bootstrap.cc:492] T 6ba071c81ec946d29ffe103da44f9ba5 P a0404a3007e14b969155f38cdde5da50: No bootstrap required, opened a new log
I20260924 16:09:42.666252 20182 ts_tablet_manager.cc:1397] T 6ba071c81ec946d29ffe103da44f9ba5 P a0404a3007e14b969155f38cdde5da50: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260924 16:09:42.666275 20180 ts_tablet_manager.cc:1397] T 6ba071c81ec946d29ffe103da44f9ba5 P 32aa91a44d354548a6a62b87606e372f: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260924 16:09:42.673442 20181 tablet_bootstrap.cc:492] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a: No bootstrap required, opened a new log
I20260924 16:09:42.673842 20181 ts_tablet_manager.cc:1397] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.009s
I20260924 16:09:42.674358 20182 raft_consensus.cc:348] T 6ba071c81ec946d29ffe103da44f9ba5 P a0404a3007e14b969155f38cdde5da50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } }
I20260924 16:09:42.674779 20182 raft_consensus.cc:374] T 6ba071c81ec946d29ffe103da44f9ba5 P a0404a3007e14b969155f38cdde5da50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:09:42.674909 20182 raft_consensus.cc:729] T 6ba071c81ec946d29ffe103da44f9ba5 P a0404a3007e14b969155f38cdde5da50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0404a3007e14b969155f38cdde5da50, State: Initialized, Role: FOLLOWER
I20260924 16:09:42.675542 20182 consensus_queue.cc:260] T 6ba071c81ec946d29ffe103da44f9ba5 P a0404a3007e14b969155f38cdde5da50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } }
I20260924 16:09:42.676672 20161 heartbeater.cc:503] Master 127.3.214.254:40829 was elected leader, sending a full tablet report...
I20260924 16:09:42.677489 20182 ts_tablet_manager.cc:1428] T 6ba071c81ec946d29ffe103da44f9ba5 P a0404a3007e14b969155f38cdde5da50: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260924 16:09:42.677781 20180 raft_consensus.cc:348] T 6ba071c81ec946d29ffe103da44f9ba5 P 32aa91a44d354548a6a62b87606e372f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } }
I20260924 16:09:42.678414 20180 raft_consensus.cc:374] T 6ba071c81ec946d29ffe103da44f9ba5 P 32aa91a44d354548a6a62b87606e372f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:09:42.678543 20180 raft_consensus.cc:729] T 6ba071c81ec946d29ffe103da44f9ba5 P 32aa91a44d354548a6a62b87606e372f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32aa91a44d354548a6a62b87606e372f, State: Initialized, Role: FOLLOWER
I20260924 16:09:42.679356 20180 consensus_queue.cc:260] T 6ba071c81ec946d29ffe103da44f9ba5 P 32aa91a44d354548a6a62b87606e372f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } }
I20260924 16:09:42.681795 20180 ts_tablet_manager.cc:1428] T 6ba071c81ec946d29ffe103da44f9ba5 P 32aa91a44d354548a6a62b87606e372f: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260924 16:09:42.686686 20181 raft_consensus.cc:348] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } }
I20260924 16:09:42.687260 20181 raft_consensus.cc:374] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:09:42.687356 20181 raft_consensus.cc:729] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d2b7a5863b74f23b9c03656247e4c3a, State: Initialized, Role: FOLLOWER
I20260924 16:09:42.688017 20181 consensus_queue.cc:260] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } }
I20260924 16:09:42.689282 20026 heartbeater.cc:503] Master 127.3.214.254:40829 was elected leader, sending a full tablet report...
I20260924 16:09:42.689839 20181 ts_tablet_manager.cc:1428] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
W20260924 16:09:42.772382 20162 tablet.cc:2369] T 6ba071c81ec946d29ffe103da44f9ba5 P a0404a3007e14b969155f38cdde5da50: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:09:42.777670 19892 tablet.cc:2369] T 6ba071c81ec946d29ffe103da44f9ba5 P 32aa91a44d354548a6a62b87606e372f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:09:42.798864 20188 raft_consensus.cc:482] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:09:42.799223 20188 raft_consensus.cc:504] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } }
I20260924 16:09:42.800989 20188 leader_election.cc:288] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0404a3007e14b969155f38cdde5da50 (127.3.214.195:42739), 32aa91a44d354548a6a62b87606e372f (127.3.214.193:39785)
I20260924 16:09:42.809844 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:42.801689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:37955 (local address 127.3.214.195:42739)
0924 16:09:42.801906 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:42.801911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:42.803090 (+ 1179us) server_negotiation.cc:407] Connection header received
0924 16:09:42.803308 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:42.803314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:42.803401 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:42.803501 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:09:42.804194 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.805181 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.807473 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.808677 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.808827 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:42.808887 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:42.809381 (+ 494us) server_negotiation.cc:299] Negotiation successful
0924 16:09:42.809542 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 16:09:42.809916 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:42.801692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:42739 (local address 127.3.214.194:37955)
0924 16:09:42.802937 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:42.802957 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:42.803145 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:42.803541 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:42.803546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:42.803584 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:09:42.804010 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:42.804022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.805333 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.805339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:42.807291 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:42.807305 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.808944 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.808951 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:42.809161 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:42.809203 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 16:09:42.809319 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 16:09:42.809488 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1049,"thread_start_us":742,"threads_started":1}
I20260924 16:09:42.810647 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:42.801653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:37823 (local address 127.3.214.193:39785)
0924 16:09:42.801978 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:42.801986 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:42.803264 (+ 1278us) server_negotiation.cc:407] Connection header received
0924 16:09:42.803555 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:42.803560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:42.803649 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:42.803770 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:09:42.804900 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.805943 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.808216 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.809257 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.809413 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:42.809490 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:42.810208 (+ 718us) server_negotiation.cc:299] Negotiation successful
0924 16:09:42.810414 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260924 16:09:42.811436 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:42.801692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:39785 (local address 127.3.214.194:37823)
0924 16:09:42.803112 (+ 1420us) negotiation.cc:107] Waiting for socket to connect
0924 16:09:42.803135 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:09:42.803391 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:09:42.804190 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:09:42.804196 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:09:42.804228 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:09:42.804687 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:42.804704 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.806187 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.806192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:42.808010 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:09:42.808030 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.809609 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.809617 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:09:42.809863 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:42.809910 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 16:09:42.810821 (+ 911us) client_negotiation.cc:240] Negotiation successful
0924 16:09:42.810992 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1251,"mutex_wait_us":227,"thread_start_us":713,"threads_started":1}
I20260924 16:09:42.811534 20115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba071c81ec946d29ffe103da44f9ba5" candidate_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0404a3007e14b969155f38cdde5da50" is_pre_election: true
I20260924 16:09:42.812083 20115 raft_consensus.cc:2454] T 6ba071c81ec946d29ffe103da44f9ba5 P a0404a3007e14b969155f38cdde5da50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d2b7a5863b74f23b9c03656247e4c3a in term 0.
I20260924 16:09:42.812733 19846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba071c81ec946d29ffe103da44f9ba5" candidate_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32aa91a44d354548a6a62b87606e372f" is_pre_election: true
I20260924 16:09:42.813028 19914 leader_election.cc:302] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d2b7a5863b74f23b9c03656247e4c3a, a0404a3007e14b969155f38cdde5da50; no voters:
I20260924 16:09:42.813282 19846 raft_consensus.cc:2454] T 6ba071c81ec946d29ffe103da44f9ba5 P 32aa91a44d354548a6a62b87606e372f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d2b7a5863b74f23b9c03656247e4c3a in term 0.
I20260924 16:09:42.813585 20188 raft_consensus.cc:2781] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:09:42.813758 20188 raft_consensus.cc:482] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:09:42.813889 20188 raft_consensus.cc:3037] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:09:42.816525 20188 raft_consensus.cc:504] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } }
I20260924 16:09:42.817363 20188 leader_election.cc:288] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a [CANDIDATE]: Term 1 election: Requested vote from peers a0404a3007e14b969155f38cdde5da50 (127.3.214.195:42739), 32aa91a44d354548a6a62b87606e372f (127.3.214.193:39785)
I20260924 16:09:42.818013 20115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba071c81ec946d29ffe103da44f9ba5" candidate_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0404a3007e14b969155f38cdde5da50"
I20260924 16:09:42.818156 19846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba071c81ec946d29ffe103da44f9ba5" candidate_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32aa91a44d354548a6a62b87606e372f"
I20260924 16:09:42.818332 20115 raft_consensus.cc:3037] T 6ba071c81ec946d29ffe103da44f9ba5 P a0404a3007e14b969155f38cdde5da50 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:09:42.818446 19846 raft_consensus.cc:3037] T 6ba071c81ec946d29ffe103da44f9ba5 P 32aa91a44d354548a6a62b87606e372f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:09:42.820793 20115 raft_consensus.cc:2454] T 6ba071c81ec946d29ffe103da44f9ba5 P a0404a3007e14b969155f38cdde5da50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d2b7a5863b74f23b9c03656247e4c3a in term 1.
I20260924 16:09:42.820991 19846 raft_consensus.cc:2454] T 6ba071c81ec946d29ffe103da44f9ba5 P 32aa91a44d354548a6a62b87606e372f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d2b7a5863b74f23b9c03656247e4c3a in term 1.
I20260924 16:09:42.821364 19914 leader_election.cc:302] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d2b7a5863b74f23b9c03656247e4c3a, a0404a3007e14b969155f38cdde5da50; no voters:
I20260924 16:09:42.821743 20188 raft_consensus.cc:2781] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:09:42.822346 20188 raft_consensus.cc:686] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a [term 1 LEADER]: Becoming Leader. State: Replica: 4d2b7a5863b74f23b9c03656247e4c3a, State: Running, Role: LEADER
I20260924 16:09:42.822944 20188 consensus_queue.cc:237] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } }
I20260924 16:09:42.829048 19701 catalog_manager.cc:5495] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d2b7a5863b74f23b9c03656247e4c3a (127.3.214.194). New cstate: current_term: 1 leader_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } health_report { overall_health: UNKNOWN } } }
I20260924 16:09:42.875780 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:42.869743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46092 (local address 127.3.214.194:40967)
0924 16:09:42.869954 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:42.869959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:42.869982 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:09:42.870318 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:42.870324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:42.870430 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:42.870599 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:09:42.871084 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.871855 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.873483 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.873957 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.874074 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:42.874651 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:09:42.875243 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:09:42.875363 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:42.875447 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 16:09:42.875527 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:09:42.886078 20115 raft_consensus.cc:1261] T 6ba071c81ec946d29ffe103da44f9ba5 P a0404a3007e14b969155f38cdde5da50 [term 1 FOLLOWER]: Refusing update from remote peer 4d2b7a5863b74f23b9c03656247e4c3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:09:42.886087 19846 raft_consensus.cc:1261] T 6ba071c81ec946d29ffe103da44f9ba5 P 32aa91a44d354548a6a62b87606e372f [term 1 FOLLOWER]: Refusing update from remote peer 4d2b7a5863b74f23b9c03656247e4c3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:09:42.887380 20188 consensus_queue.cc:1035] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:09:42.887740 20191 consensus_queue.cc:1035] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:09:42.923460 19700 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:09:42.923797 19700 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:09:42.928735 19700 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7: Sending DeleteTablet for 3 replicas of tablet 6ba071c81ec946d29ffe103da44f9ba5
I20260924 16:09:42.930199 19961 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ba071c81ec946d29ffe103da44f9ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:09:42 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:46082
I20260924 16:09:42.930482 19826 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ba071c81ec946d29ffe103da44f9ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:09:42 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:59778
I20260924 16:09:42.931619 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 19763
I20260924 16:09:42.933274 20205 tablet_replica.cc:330] T 6ba071c81ec946d29ffe103da44f9ba5 P 32aa91a44d354548a6a62b87606e372f: stopping tablet replica
I20260924 16:09:42.933647 20095 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ba071c81ec946d29ffe103da44f9ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:09:42 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:52214
I20260924 16:09:42.934005 20205 raft_consensus.cc:2229] T 6ba071c81ec946d29ffe103da44f9ba5 P 32aa91a44d354548a6a62b87606e372f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:09:42.934672 20205 raft_consensus.cc:2258] T 6ba071c81ec946d29ffe103da44f9ba5 P 32aa91a44d354548a6a62b87606e372f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:09:42.935652 20207 tablet_replica.cc:330] T 6ba071c81ec946d29ffe103da44f9ba5 P a0404a3007e14b969155f38cdde5da50: stopping tablet replica
I20260924 16:09:42.936239 20207 raft_consensus.cc:2229] T 6ba071c81ec946d29ffe103da44f9ba5 P a0404a3007e14b969155f38cdde5da50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:09:42.936921 20207 raft_consensus.cc:2258] T 6ba071c81ec946d29ffe103da44f9ba5 P a0404a3007e14b969155f38cdde5da50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:09:42.937835 20205 ts_tablet_manager.cc:1905] T 6ba071c81ec946d29ffe103da44f9ba5 P 32aa91a44d354548a6a62b87606e372f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:09:42.939471 20207 ts_tablet_manager.cc:1905] T 6ba071c81ec946d29ffe103da44f9ba5 P a0404a3007e14b969155f38cdde5da50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:09:42.940428 20204 tablet_replica.cc:330] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a: stopping tablet replica
I20260924 16:09:42.940819 20204 raft_consensus.cc:2229] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:09:42.941581 20204 raft_consensus.cc:2258] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:09:42.943796 20204 ts_tablet_manager.cc:1905] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:09:42.948307 20205 ts_tablet_manager.cc:1918] T 6ba071c81ec946d29ffe103da44f9ba5 P 32aa91a44d354548a6a62b87606e372f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:09:42.948720 20205 log.cc:1196] T 6ba071c81ec946d29ffe103da44f9ba5 P 32aa91a44d354548a6a62b87606e372f: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/wal/wals/6ba071c81ec946d29ffe103da44f9ba5
I20260924 16:09:42.949549 20205 ts_tablet_manager.cc:1939] T 6ba071c81ec946d29ffe103da44f9ba5 P 32aa91a44d354548a6a62b87606e372f: Deleting consensus metadata
I20260924 16:09:42.952904 19688 catalog_manager.cc:4841] TS 32aa91a44d354548a6a62b87606e372f (127.3.214.193:39785): tablet 6ba071c81ec946d29ffe103da44f9ba5 (table test-table [id=43282d023bf34113ab57072968775a5d]) successfully deleted
I20260924 16:09:42.956593 20207 ts_tablet_manager.cc:1918] T 6ba071c81ec946d29ffe103da44f9ba5 P a0404a3007e14b969155f38cdde5da50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:09:42.956985 20207 log.cc:1196] T 6ba071c81ec946d29ffe103da44f9ba5 P a0404a3007e14b969155f38cdde5da50: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/6ba071c81ec946d29ffe103da44f9ba5
I20260924 16:09:42.957324 20204 ts_tablet_manager.cc:1918] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:09:42.957687 20204 log.cc:1196] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/6ba071c81ec946d29ffe103da44f9ba5
I20260924 16:09:42.957718 20207 ts_tablet_manager.cc:1939] T 6ba071c81ec946d29ffe103da44f9ba5 P a0404a3007e14b969155f38cdde5da50: Deleting consensus metadata
I20260924 16:09:42.958541 20204 ts_tablet_manager.cc:1939] T 6ba071c81ec946d29ffe103da44f9ba5 P 4d2b7a5863b74f23b9c03656247e4c3a: Deleting consensus metadata
I20260924 16:09:42.960387 19686 catalog_manager.cc:4841] TS a0404a3007e14b969155f38cdde5da50 (127.3.214.195:42739): tablet 6ba071c81ec946d29ffe103da44f9ba5 (table test-table [id=43282d023bf34113ab57072968775a5d]) successfully deleted
I20260924 16:09:42.961447 19686 catalog_manager.cc:4841] TS 4d2b7a5863b74f23b9c03656247e4c3a (127.3.214.194:40967): tablet 6ba071c81ec946d29ffe103da44f9ba5 (table test-table [id=43282d023bf34113ab57072968775a5d]) successfully deleted
Sep 24 16:09:42 dist-test-slave-1gc3 krb5kdc[19656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.193@KRBTEST.COM
I20260924 16:09:42.982429 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:42.933235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59786 (local address 127.3.214.193:39785)
0924 16:09:42.933495 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:42.933503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:42.962075 (+ 28572us) server_negotiation.cc:407] Connection header received
0924 16:09:42.962375 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:42.962379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:42.962476 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:42.962595 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:42.964791 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.965659 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.967733 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:42.968221 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:42.968399 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:42.974335 (+ 5936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:42.974364 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:42.974392 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:42.974456 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:42.978171 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:42.979458 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:42.979470 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:42.979473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:42.979678 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:42.980206 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:42.980212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:42.980215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:42.980784 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:42.981091 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:42.981617 (+ 526us) server_negotiation.cc:299] Negotiation successful
0924 16:09:42.981879 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260924 16:09:42.983428 19886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:43.085489 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 19763
I20260924 16:09:43.106405 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 19897
Sep 24 16:09:43 dist-test-slave-1gc3 krb5kdc[19656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.194@KRBTEST.COM
I20260924 16:09:43.127035 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:43.107331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46106 (local address 127.3.214.194:40967)
0924 16:09:43.107541 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:43.107552 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:43.107804 (+ 252us) server_negotiation.cc:407] Connection header received
0924 16:09:43.108021 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:43.108026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:43.108121 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:43.108218 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:43.110060 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:43.111192 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:43.113216 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:43.113721 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:43.113818 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:43.120061 (+ 6243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:43.120089 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:43.120115 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:43.120182 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:43.123641 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:43.124575 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:43.124583 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:43.124585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:43.124757 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:43.125148 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:43.125152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:43.125154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:43.125604 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:43.125950 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:43.126399 (+ 449us) server_negotiation.cc:299] Negotiation successful
0924 16:09:43.126645 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:09:43.127797 20021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:43.261950 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 19897
I20260924 16:09:43.281693 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 20032
Sep 24 16:09:43 dist-test-slave-1gc3 krb5kdc[19656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.195@KRBTEST.COM
I20260924 16:09:43.301625 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:09:43.282508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52216 (local address 127.3.214.195:42739)
0924 16:09:43.282715 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 16:09:43.282722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:09:43.282840 (+ 118us) server_negotiation.cc:407] Connection header received
0924 16:09:43.283057 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:09:43.283062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:09:43.283138 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:09:43.283226 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:09:43.284528 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:43.285251 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:43.286638 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:09:43.287095 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:09:43.287206 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:09:43.295520 (+ 8314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:09:43.295541 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:09:43.295562 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:09:43.295618 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:09:43.298340 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:43.299160 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:43.299169 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:43.299172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:43.299341 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:09:43.299735 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:09:43.299738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:09:43.299741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:09:43.300313 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:09:43.300625 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 16:09:43.301012 (+ 387us) server_negotiation.cc:299] Negotiation successful
0924 16:09:43.301261 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:09:43.302331 20156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:43.405128 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 20032
I20260924 16:09:43.425091 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 19669
I20260924 16:09:43.426681 19731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:09:43.544579 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 19669
I20260924 16:10:03.570163 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:40829
--webserver_interface=127.3.214.254
--webserver_port=38721
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:40829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:10:03.939182 20212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:10:03.939524 20212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:10:03.939632 20212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:10:03.947610 20212 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 16:10:03.947739 20212 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 16:10:03.947779 20212 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 16:10:03.947832 20212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:10:03.947912 20212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:10:03.948011 20212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:10:03.948071 20212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:10:03.948184 20212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:10:03.958923 20212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:40829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:40829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=38721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:10:03.960354 20212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:10:03.961748 20212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:10:03.989562 20218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:03.989475 20219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:03.989972 20212 server_base.cc:1034] running on GCE node
W20260924 16:10:03.989564 20221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:03.991178 20212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:10:03.993041 20212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:10:03.994428 20212 hybrid_clock.cc:648] HybridClock initialized: now 1790266203994290 us; error 87 us; skew 500 ppm
I20260924 16:10:03.995023 20212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:10:03 dist-test-slave-1gc3 krb5kdc[19656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266203, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:10:04.000196 20212 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:10:04.003023 20212 webserver.cc:460] Webserver started at http://127.3.214.254:38721/ using document root <none> and password file <none>
I20260924 16:10:04.003999 20212 fs_manager.cc:359] Metadata directory not provided
I20260924 16:10:04.004150 20212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:10:04.011615 20212 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260924 16:10:04.015791 20228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:04.017808 20212 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:10:04.018080 20212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "2ae01576d2fc4609972bb444423d12a7"
format_stamp: "Formatted at 2026-09-24 16:09:40 on dist-test-slave-1gc3"
server_key: "75d417bd5c0bf24aac1c7a7c34763393"
server_key_iv: "72257ba305e7e54adba06fc624aad30e"
server_key_version: "encryptionkey@0"
I20260924 16:10:04.019385 20212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:10:04.034540 20212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:10:04.040122 20212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:10:04.040478 20212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:10:04.065095 20212 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:40829
I20260924 16:10:04.065186 20279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:40829 every 8 connection(s)
I20260924 16:10:04.067103 20212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:10:04.076859 20280 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:10:04.077519 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 20212
I20260924 16:10:04.078440 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:39785
--local_ip_for_outbound_sockets=127.3.214.193
--tserver_master_addrs=127.3.214.254:40829
--webserver_port=43921
--webserver_interface=127.3.214.193
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:10:04.081497 20280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7: Bootstrap starting.
I20260924 16:10:04.111318 20280 log.cc:824] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7: Log is configured to *not* fsync() on all Append() calls
I20260924 16:10:04.127347 20280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:10:04.127981 20280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7: Bootstrap complete.
I20260924 16:10:04.138696 20280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae01576d2fc4609972bb444423d12a7" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40829 } }
I20260924 16:10:04.139698 20280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae01576d2fc4609972bb444423d12a7, State: Initialized, Role: FOLLOWER
I20260924 16:10:04.140424 20280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae01576d2fc4609972bb444423d12a7" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40829 } }
I20260924 16:10:04.140676 20280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:10:04.140789 20280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:10:04.140952 20280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:10:04.145287 20280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae01576d2fc4609972bb444423d12a7" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40829 } }
I20260924 16:10:04.145885 20280 leader_election.cc:302] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ae01576d2fc4609972bb444423d12a7; no voters:
I20260924 16:10:04.146608 20280 leader_election.cc:288] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 16:10:04.146752 20284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 16:10:04.148304 20284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 2 LEADER]: Becoming Leader. State: Replica: 2ae01576d2fc4609972bb444423d12a7, State: Running, Role: LEADER
I20260924 16:10:04.149058 20284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae01576d2fc4609972bb444423d12a7" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40829 } }
I20260924 16:10:04.149385 20280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:10:04.152119 20286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ae01576d2fc4609972bb444423d12a7. Latest consensus state: current_term: 2 leader_uuid: "2ae01576d2fc4609972bb444423d12a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae01576d2fc4609972bb444423d12a7" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40829 } } }
I20260924 16:10:04.152611 20286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:10:04.153223 20285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2ae01576d2fc4609972bb444423d12a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae01576d2fc4609972bb444423d12a7" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40829 } } }
I20260924 16:10:04.153553 20285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:10:04.160028 20290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:10:04.171157 20290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:10:04.172959 20290 catalog_manager.cc:1259] Loaded cluster ID: e809d8a952f6463ab15ca057e2e792a5
I20260924 16:10:04.173087 20290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:10:04.177946 20290 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:10:04.180240 20290 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7: Loaded TSK: 0
I20260924 16:10:04.229380 20290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7: Generated new TSK 1
I20260924 16:10:04.229960 20290 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:10:04.520470 20282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:10:04.520833 20282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:10:04.520959 20282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:10:04.529254 20282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:10:04.529402 20282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:10:04.529560 20282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:10:04.541020 20282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:39785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=43921
--tserver_master_addrs=127.3.214.254:40829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:10:04.542862 20282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:10:04.544589 20282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:10:04.563424 20308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:04.564471 20307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:04.563683 20311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:04.566581 20282 server_base.cc:1034] running on GCE node
I20260924 16:10:04.567725 20282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:10:04.569651 20282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:10:04.570976 20282 hybrid_clock.cc:648] HybridClock initialized: now 1790266204570874 us; error 87 us; skew 500 ppm
I20260924 16:10:04.571475 20282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:10:04 dist-test-slave-1gc3 krb5kdc[19656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266204, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:10:04.575270 20282 init.cc:377] Logged in from keytab as kudu/127.3.214.193@KRBTEST.COM (short username kudu)
I20260924 16:10:04.577073 20282 webserver.cc:460] Webserver started at http://127.3.214.193:43921/ using document root <none> and password file <none>
I20260924 16:10:04.577924 20282 fs_manager.cc:359] Metadata directory not provided
I20260924 16:10:04.578064 20282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:10:04.584475 20282 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:10:04.588263 20318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:04.590046 20282 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260924 16:10:04.590349 20282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "32aa91a44d354548a6a62b87606e372f"
format_stamp: "Formatted at 2026-09-24 16:09:41 on dist-test-slave-1gc3"
server_key: "9c78082edc6a91d8166f6c3e36eceab8"
server_key_iv: "cd71c2897678fbb9056137e0c837911e"
server_key_version: "encryptionkey@0"
I20260924 16:10:04.591846 20282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:10:04.612422 20282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:10:04.617151 20282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:10:04.617441 20282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:10:04.618713 20282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:10:04.620467 20282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:10:04.620576 20282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:04.620700 20282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:10:04.620770 20282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:04.657476 20282 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:39785
I20260924 16:10:04.657567 20430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:39785 every 8 connection(s)
I20260924 16:10:04.659219 20282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:10:04.660461 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 20282
I20260924 16:10:04.661090 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:40967
--local_ip_for_outbound_sockets=127.3.214.194
--tserver_master_addrs=127.3.214.254:40829
--webserver_port=45585
--webserver_interface=127.3.214.194
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:10:04 dist-test-slave-1gc3 krb5kdc[19656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266204, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:10:04.691373 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:04.664492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:47403 (local address 127.3.214.254:40829)
0924 16:10:04.666671 (+ 2179us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:04.666690 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:04.666746 (+ 56us) server_negotiation.cc:407] Connection header received
0924 16:10:04.668101 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:04.668124 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:04.668654 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:04.669129 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:04.671435 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:04.673037 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:04.675786 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:04.676519 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:04.676685 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:04.681945 (+ 5260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:04.681955 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:04.681977 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:04.682030 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:04.685359 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:04.686821 (+ 1462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:04.686829 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:04.686838 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:04.687051 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:04.687660 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:04.687663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:04.687665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:04.688312 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:04.688488 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:04.689830 (+ 1342us) server_negotiation.cc:299] Negotiation successful
0924 16:10:04.690284 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1889,"thread_start_us":1748,"threads_started":1}
I20260924 16:10:04.692196 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:04.663421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.193:47403)
0924 16:10:04.664633 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:04.664714 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:04.667365 (+ 2651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:04.669454 (+ 2089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:04.669475 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:04.670350 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:10:04.671120 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:04.671144 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:04.673300 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:04.673308 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:04.675563 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:04.675587 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:04.676826 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:04.676835 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:04.677166 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:04.678023 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:10:04.678057 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:10:04.681502 (+ 3445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:10:04.685624 (+ 4122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:10:04.685639 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:10:04.685667 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:10:04.686608 (+ 941us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:10:04.687182 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:10:04.687188 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:10:04.687191 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:10:04.687530 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:10:04.688511 (+ 981us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:10:04.688525 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:10:04.689183 (+ 658us) client_negotiation.cc:769] Sending connection context
0924 16:10:04.689634 (+ 451us) client_negotiation.cc:240] Negotiation successful
0924 16:10:04.690464 (+ 830us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":556,"threads_started":1}
I20260924 16:10:04.695323 20431 heartbeater.cc:347] Connected to a master server at 127.3.214.254:40829
I20260924 16:10:04.695865 20431 heartbeater.cc:464] Registering TS with master...
I20260924 16:10:04.697244 20431 heartbeater.cc:511] Master 127.3.214.254:40829 requested a full tablet report, sending...
I20260924 16:10:04.700147 20245 ts_manager.cc:194] Registered new tserver with Master: 32aa91a44d354548a6a62b87606e372f (127.3.214.193:39785)
I20260924 16:10:04.701769 20245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:47403
W20260924 16:10:05.090551 20435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:10:05.090945 20435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:10:05.091066 20435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:10:05.100426 20435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:10:05.100620 20435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:10:05.100812 20435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:10:05.112599 20435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:40967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=45585
--tserver_master_addrs=127.3.214.254:40829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:10:05.114601 20435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:10:05.116423 20435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:10:05.134755 20441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:05.134856 20440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:05.134819 20443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:05.135919 20435 server_base.cc:1034] running on GCE node
I20260924 16:10:05.136875 20435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:10:05.138409 20435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:10:05.139720 20435 hybrid_clock.cc:648] HybridClock initialized: now 1790266205139649 us; error 54 us; skew 500 ppm
I20260924 16:10:05.140225 20435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:10:05 dist-test-slave-1gc3 krb5kdc[19656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266205, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:10:05.145161 20435 init.cc:377] Logged in from keytab as kudu/127.3.214.194@KRBTEST.COM (short username kudu)
I20260924 16:10:05.147871 20435 webserver.cc:460] Webserver started at http://127.3.214.194:45585/ using document root <none> and password file <none>
I20260924 16:10:05.149360 20435 fs_manager.cc:359] Metadata directory not provided
I20260924 16:10:05.149596 20435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:10:05.157728 20435 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 16:10:05.162425 20450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:05.164992 20435 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260924 16:10:05.165302 20435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "4d2b7a5863b74f23b9c03656247e4c3a"
format_stamp: "Formatted at 2026-09-24 16:09:41 on dist-test-slave-1gc3"
server_key: "efdacfceba9f987c930af1dbdfc25b85"
server_key_iv: "cecbc330ff8ac6164e4a2812d8ffda9a"
server_key_version: "encryptionkey@0"
I20260924 16:10:05.166955 20435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:10:05.223434 20435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:10:05.229859 20435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:10:05.230262 20435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:10:05.231673 20435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:10:05.233660 20435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:10:05.233793 20435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:05.233930 20435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:10:05.234007 20435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:05.275362 20435 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:40967
I20260924 16:10:05.275440 20564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:40967 every 8 connection(s)
I20260924 16:10:05.277257 20435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:10:05.281975 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 20435
I20260924 16:10:05.283027 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:42739
--local_ip_for_outbound_sockets=127.3.214.195
--tserver_master_addrs=127.3.214.254:40829
--webserver_port=41353
--webserver_interface=127.3.214.195
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:10:05 dist-test-slave-1gc3 krb5kdc[19656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266205, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:10:05.304215 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:05.280757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:42823 (local address 127.3.214.254:40829)
0924 16:10:05.281265 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:05.281269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:05.283087 (+ 1818us) server_negotiation.cc:407] Connection header received
0924 16:10:05.286185 (+ 3098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:05.286194 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:05.286313 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:05.286462 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:05.289031 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.289865 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.292411 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.292977 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.293120 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:05.297500 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:05.297506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:05.297510 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:05.297555 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:05.300632 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:05.301552 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:05.301556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:05.301559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:05.301712 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:05.302208 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:05.302212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:05.302214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:05.302527 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:05.302683 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:05.303519 (+ 836us) server_negotiation.cc:299] Negotiation successful
0924 16:10:05.303796 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":260,"threads_started":1}
I20260924 16:10:05.305383 20568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:05.281443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.194:42823)
0924 16:10:05.282777 (+ 1334us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:05.282859 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:05.285632 (+ 2773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:05.286866 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:05.286880 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:05.287773 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:10:05.288728 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:05.288761 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.290158 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.290164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:05.292121 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:05.292146 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.293304 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.293313 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:05.293660 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:05.294487 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:10:05.294520 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:10:05.297154 (+ 2634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:10:05.300848 (+ 3694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:10:05.300858 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:10:05.300877 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:10:05.301396 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:10:05.301842 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:10:05.301848 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:10:05.301851 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:10:05.302066 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:10:05.302700 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:10:05.302711 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:10:05.303121 (+ 410us) client_negotiation.cc:769] Sending connection context
0924 16:10:05.303395 (+ 274us) client_negotiation.cc:240] Negotiation successful
0924 16:10:05.304031 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":392,"threads_started":1}
I20260924 16:10:05.307701 20565 heartbeater.cc:347] Connected to a master server at 127.3.214.254:40829
I20260924 16:10:05.308136 20565 heartbeater.cc:464] Registering TS with master...
I20260924 16:10:05.309069 20565 heartbeater.cc:511] Master 127.3.214.254:40829 requested a full tablet report, sending...
I20260924 16:10:05.311120 20245 ts_manager.cc:194] Registered new tserver with Master: 4d2b7a5863b74f23b9c03656247e4c3a (127.3.214.194:40967)
I20260924 16:10:05.311990 20245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:42823
I20260924 16:10:05.707078 20431 heartbeater.cc:503] Master 127.3.214.254:40829 was elected leader, sending a full tablet report...
W20260924 16:10:05.728757 20569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:10:05.729192 20569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:10:05.729308 20569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:10:05.737604 20569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:10:05.737779 20569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:10:05.737913 20569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:10:05.749511 20569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:42739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=41353
--tserver_master_addrs=127.3.214.254:40829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:10:05.751395 20569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:10:05.753043 20569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:10:05.771919 20576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:05.771911 20575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:05.771932 20578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:05.773334 20569 server_base.cc:1034] running on GCE node
I20260924 16:10:05.774425 20569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:10:05.775991 20569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:10:05.777453 20569 hybrid_clock.cc:648] HybridClock initialized: now 1790266205777353 us; error 86 us; skew 500 ppm
I20260924 16:10:05.777989 20569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:10:05 dist-test-slave-1gc3 krb5kdc[19656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266205, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:10:05.783157 20569 init.cc:377] Logged in from keytab as kudu/127.3.214.195@KRBTEST.COM (short username kudu)
I20260924 16:10:05.785481 20569 webserver.cc:460] Webserver started at http://127.3.214.195:41353/ using document root <none> and password file <none>
I20260924 16:10:05.786620 20569 fs_manager.cc:359] Metadata directory not provided
I20260924 16:10:05.786805 20569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:10:05.795360 20569 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 16:10:05.800016 20585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:05.802083 20569 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260924 16:10:05.802423 20569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "a0404a3007e14b969155f38cdde5da50"
format_stamp: "Formatted at 2026-09-24 16:09:42 on dist-test-slave-1gc3"
server_key: "ed460bb15d3791f3fed5f64e0c007bb2"
server_key_iv: "223e26642e659b0bf4f3473ad35d25dc"
server_key_version: "encryptionkey@0"
I20260924 16:10:05.803784 20569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:10:05.822386 20569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:10:05.828554 20569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:10:05.828888 20569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:10:05.830284 20569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:10:05.832187 20569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:10:05.832314 20569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:05.832418 20569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:10:05.832495 20569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:05.872195 20569 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:42739
I20260924 16:10:05.872159 20699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:42739 every 8 connection(s)
I20260924 16:10:05.874532 20569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:10:05.882278 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 20569
Sep 24 16:10:05 dist-test-slave-1gc3 krb5kdc[19656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266205, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:10:05.901691 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:05.883396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60938 (local address 127.3.214.254:40829)
0924 16:10:05.884300 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:05.884305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:05.884324 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:10:05.884468 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:05.884471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:05.884561 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:05.884654 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:05.886812 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.887717 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.889592 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.890028 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.890208 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:05.894231 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:05.894239 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:05.894244 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:05.894293 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:05.898394 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:05.899618 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:05.899623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:05.899626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:05.899804 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:05.900268 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:05.900272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:05.900274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:05.900572 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:05.900839 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:05.901189 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 16:10:05.901391 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":645,"threads_started":1}
I20260924 16:10:05.903210 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:05.877956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:56615 (local address 127.3.214.254:40829)
0924 16:10:05.878622 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:05.878629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:05.879829 (+ 1200us) server_negotiation.cc:407] Connection header received
0924 16:10:05.882304 (+ 2475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:05.882311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:05.882452 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:05.882656 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:05.885383 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.886294 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.888612 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.889084 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.889191 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:05.894893 (+ 5702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:05.894901 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:05.894905 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:05.894978 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:05.898743 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:05.900001 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:05.900005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:05.900008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:05.900144 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:05.900689 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:05.900694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:05.900697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:05.900982 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:05.901138 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:05.902580 (+ 1442us) server_negotiation.cc:299] Negotiation successful
0924 16:10:05.902830 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":344,"threads_started":1}
I20260924 16:10:05.904753 20703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:05.878556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.195:56615)
0924 16:10:05.879559 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:05.879624 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:05.881831 (+ 2207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:05.883003 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:05.883015 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:05.883976 (+ 961us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:10:05.885108 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:05.885153 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.886525 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.886530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:05.888408 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:05.888428 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.889377 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.889386 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:05.889725 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:05.890686 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:10:05.890726 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:10:05.894372 (+ 3646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:10:05.899002 (+ 4630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:10:05.899018 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:10:05.899049 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:10:05.899850 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:10:05.900275 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:10:05.900279 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:10:05.900282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:10:05.900558 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:10:05.901146 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:10:05.901157 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:10:05.901953 (+ 796us) client_negotiation.cc:769] Sending connection context
0924 16:10:05.902428 (+ 475us) client_negotiation.cc:240] Negotiation successful
0924 16:10:05.903150 (+ 722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":366,"threads_started":1}
I20260924 16:10:05.906641 20700 heartbeater.cc:347] Connected to a master server at 127.3.214.254:40829
I20260924 16:10:05.907085 20700 heartbeater.cc:464] Registering TS with master...
I20260924 16:10:05.908092 20700 heartbeater.cc:511] Master 127.3.214.254:40829 requested a full tablet report, sending...
I20260924 16:10:05.910288 20245 ts_manager.cc:194] Registered new tserver with Master: a0404a3007e14b969155f38cdde5da50 (127.3.214.195:42739)
I20260924 16:10:05.911283 20245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:56615
I20260924 16:10:05.919317 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:10:05.927182 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:05.920652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60940 (local address 127.3.214.254:40829)
0924 16:10:05.920861 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:05.920867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:05.921400 (+ 533us) server_negotiation.cc:407] Connection header received
0924 16:10:05.921688 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:05.921692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:05.921860 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:05.922024 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:10:05.922568 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.923413 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.925025 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.925473 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.925602 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:05.926223 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:10:05.926464 (+ 241us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 16:10:05.926865 (+ 401us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60940: token expired
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 16:10:05.927320 20702 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60940: token expired
W20260924 16:10:05.928164 20167 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.214.254:40829)
I20260924 16:10:05.945099 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:05.928863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60948 (local address 127.3.214.254:40829)
0924 16:10:05.929116 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:05.929123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:05.929148 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:10:05.929307 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:05.929313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:05.929394 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:05.929485 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:05.931130 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.932035 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.933616 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.934198 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.934326 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:05.937593 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:05.937600 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:05.937603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:05.937655 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:05.941261 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:05.942379 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:05.942384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:05.942387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:05.942547 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:05.943196 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:05.943200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:05.943203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:05.943687 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:05.943979 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:05.944440 (+ 461us) server_negotiation.cc:299] Negotiation successful
0924 16:10:05.944684 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 16:10:05.949981 20245 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:10:05.959858 20245 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:10:06.000279 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:05.976269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55768 (local address 127.3.214.195:42739)
0924 16:10:05.990352 (+ 14083us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:05.990361 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:05.990400 (+ 39us) server_negotiation.cc:407] Connection header received
0924 16:10:05.990496 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:05.990502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:05.990849 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:05.991098 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:10:05.992472 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.994044 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.997346 (+ 3302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.998774 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.998916 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:05.999211 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:05.999613 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 16:10:05.999871 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13876,"thread_start_us":13650,"threads_started":1}
I20260924 16:10:06.001379 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:05.975781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:42739 (local address 127.0.0.1:55768)
0924 16:10:05.988960 (+ 13179us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:05.989053 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:05.989421 (+ 368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:05.991480 (+ 2059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:05.991486 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:05.991526 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:10:05.992228 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:05.992242 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.994221 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.994227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:05.997118 (+ 2891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:05.997138 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.999023 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.999029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:05.999277 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:05.999342 (+ 65us) client_negotiation.cc:769] Sending connection context
0924 16:10:06.000542 (+ 1200us) client_negotiation.cc:240] Negotiation successful
0924 16:10:06.000732 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12698,"mutex_wait_us":126,"thread_start_us":13686,"threads_started":1}
I20260924 16:10:06.001413 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:05.976455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46134 (local address 127.3.214.193:39785)
0924 16:10:05.987528 (+ 11073us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:05.987546 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:05.990008 (+ 2462us) server_negotiation.cc:407] Connection header received
0924 16:10:05.990481 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:05.990493 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:05.990850 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:05.991099 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:10:05.992281 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.993862 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.997655 (+ 3793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.999318 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.999504 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:05.999791 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:06.000878 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0924 16:10:06.001084 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10594,"thread_start_us":11146,"threads_started":1}
I20260924 16:10:06.001505 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:05.991045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:40967 (local address 127.0.0.1:55282)
0924 16:10:05.992006 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:05.992032 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:05.992181 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:05.993055 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:05.993060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:05.993092 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:10:05.993465 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:05.993479 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.995114 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.995120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:05.997689 (+ 2569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:05.997705 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:06.000007 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:06.000480 (+ 473us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:06.000674 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:06.000706 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 16:10:06.000795 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 16:10:06.000952 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":25,"thread_start_us":507,"threads_started":1}
I20260924 16:10:06.001978 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:05.976088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:39785 (local address 127.0.0.1:46134)
0924 16:10:05.989840 (+ 13752us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:05.989861 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:05.990214 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:05.991147 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:05.991156 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:05.991195 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:10:05.991964 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:05.991986 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.993990 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.993995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:05.997470 (+ 3475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:05.997481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:06.000043 (+ 2562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:06.000050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:06.000243 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:06.000281 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 16:10:06.000384 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 16:10:06.000665 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13598,"mutex_wait_us":161,"thread_start_us":12181,"threads_started":1}
I20260924 16:10:06.002352 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:05.990365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55282 (local address 127.3.214.194:40967)
0924 16:10:05.990952 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:05.990962 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:05.992347 (+ 1385us) server_negotiation.cc:407] Connection header received
0924 16:10:05.992491 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:05.992500 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:05.992795 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:05.993025 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:10:05.993649 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.994903 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.997925 (+ 3022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:05.999400 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:05.999537 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:05.999757 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:06.001774 (+ 2017us) server_negotiation.cc:299] Negotiation successful
0924 16:10:06.001975 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":312,"threads_started":1}
I20260924 16:10:06.007151 20498 tablet_service.cc:1467] Processing CreateTablet for tablet 6ad2e80ac76d4833948d0b144469ffeb (DEFAULT_TABLE table=test-table [id=6e4392ca6c04441a89bb016cae319e8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:10:06.007151 20633 tablet_service.cc:1467] Processing CreateTablet for tablet 6ad2e80ac76d4833948d0b144469ffeb (DEFAULT_TABLE table=test-table [id=6e4392ca6c04441a89bb016cae319e8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:10:06.009131 20633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ad2e80ac76d4833948d0b144469ffeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:10:06.009131 20498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ad2e80ac76d4833948d0b144469ffeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:10:06.009651 20366 tablet_service.cc:1467] Processing CreateTablet for tablet 6ad2e80ac76d4833948d0b144469ffeb (DEFAULT_TABLE table=test-table [id=6e4392ca6c04441a89bb016cae319e8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:10:06.011322 20366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ad2e80ac76d4833948d0b144469ffeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:10:06.026597 20717 tablet_bootstrap.cc:492] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f: Bootstrap starting.
I20260924 16:10:06.030050 20719 tablet_bootstrap.cc:492] T 6ad2e80ac76d4833948d0b144469ffeb P a0404a3007e14b969155f38cdde5da50: Bootstrap starting.
I20260924 16:10:06.031850 20717 tablet_bootstrap.cc:654] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f: Neither blocks nor log segments found. Creating new log.
I20260924 16:10:06.033313 20717 log.cc:824] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:10:06.034304 20719 tablet_bootstrap.cc:654] T 6ad2e80ac76d4833948d0b144469ffeb P a0404a3007e14b969155f38cdde5da50: Neither blocks nor log segments found. Creating new log.
I20260924 16:10:06.034397 20718 tablet_bootstrap.cc:492] T 6ad2e80ac76d4833948d0b144469ffeb P 4d2b7a5863b74f23b9c03656247e4c3a: Bootstrap starting.
I20260924 16:10:06.036123 20719 log.cc:824] T 6ad2e80ac76d4833948d0b144469ffeb P a0404a3007e14b969155f38cdde5da50: Log is configured to *not* fsync() on all Append() calls
I20260924 16:10:06.038133 20717 tablet_bootstrap.cc:492] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f: No bootstrap required, opened a new log
I20260924 16:10:06.038554 20717 ts_tablet_manager.cc:1397] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260924 16:10:06.039242 20718 tablet_bootstrap.cc:654] T 6ad2e80ac76d4833948d0b144469ffeb P 4d2b7a5863b74f23b9c03656247e4c3a: Neither blocks nor log segments found. Creating new log.
I20260924 16:10:06.040083 20719 tablet_bootstrap.cc:492] T 6ad2e80ac76d4833948d0b144469ffeb P a0404a3007e14b969155f38cdde5da50: No bootstrap required, opened a new log
I20260924 16:10:06.040388 20719 ts_tablet_manager.cc:1397] T 6ad2e80ac76d4833948d0b144469ffeb P a0404a3007e14b969155f38cdde5da50: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260924 16:10:06.040799 20718 log.cc:824] T 6ad2e80ac76d4833948d0b144469ffeb P 4d2b7a5863b74f23b9c03656247e4c3a: Log is configured to *not* fsync() on all Append() calls
I20260924 16:10:06.045353 20718 tablet_bootstrap.cc:492] T 6ad2e80ac76d4833948d0b144469ffeb P 4d2b7a5863b74f23b9c03656247e4c3a: No bootstrap required, opened a new log
I20260924 16:10:06.045745 20718 ts_tablet_manager.cc:1397] T 6ad2e80ac76d4833948d0b144469ffeb P 4d2b7a5863b74f23b9c03656247e4c3a: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260924 16:10:06.049072 20719 raft_consensus.cc:348] T 6ad2e80ac76d4833948d0b144469ffeb P a0404a3007e14b969155f38cdde5da50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } }
I20260924 16:10:06.049543 20719 raft_consensus.cc:374] T 6ad2e80ac76d4833948d0b144469ffeb P a0404a3007e14b969155f38cdde5da50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:10:06.049636 20719 raft_consensus.cc:729] T 6ad2e80ac76d4833948d0b144469ffeb P a0404a3007e14b969155f38cdde5da50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0404a3007e14b969155f38cdde5da50, State: Initialized, Role: FOLLOWER
I20260924 16:10:06.049746 20717 raft_consensus.cc:348] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } }
I20260924 16:10:06.050369 20717 raft_consensus.cc:374] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:10:06.050495 20717 raft_consensus.cc:729] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32aa91a44d354548a6a62b87606e372f, State: Initialized, Role: FOLLOWER
I20260924 16:10:06.050405 20719 consensus_queue.cc:260] T 6ad2e80ac76d4833948d0b144469ffeb P a0404a3007e14b969155f38cdde5da50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } }
I20260924 16:10:06.051270 20717 consensus_queue.cc:260] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } }
I20260924 16:10:06.055366 20717 ts_tablet_manager.cc:1428] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260924 16:10:06.057179 20718 raft_consensus.cc:348] T 6ad2e80ac76d4833948d0b144469ffeb P 4d2b7a5863b74f23b9c03656247e4c3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } }
I20260924 16:10:06.057803 20718 raft_consensus.cc:374] T 6ad2e80ac76d4833948d0b144469ffeb P 4d2b7a5863b74f23b9c03656247e4c3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:10:06.057904 20718 raft_consensus.cc:729] T 6ad2e80ac76d4833948d0b144469ffeb P 4d2b7a5863b74f23b9c03656247e4c3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d2b7a5863b74f23b9c03656247e4c3a, State: Initialized, Role: FOLLOWER
I20260924 16:10:06.058686 20718 consensus_queue.cc:260] T 6ad2e80ac76d4833948d0b144469ffeb P 4d2b7a5863b74f23b9c03656247e4c3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } }
I20260924 16:10:06.060184 20700 heartbeater.cc:503] Master 127.3.214.254:40829 was elected leader, sending a full tablet report...
I20260924 16:10:06.060943 20719 ts_tablet_manager.cc:1428] T 6ad2e80ac76d4833948d0b144469ffeb P a0404a3007e14b969155f38cdde5da50: Time spent starting tablet: real 0.020s user 0.012s sys 0.009s
I20260924 16:10:06.061362 20718 ts_tablet_manager.cc:1428] T 6ad2e80ac76d4833948d0b144469ffeb P 4d2b7a5863b74f23b9c03656247e4c3a: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260924 16:10:06.061959 20565 heartbeater.cc:503] Master 127.3.214.254:40829 was elected leader, sending a full tablet report...
W20260924 16:10:06.126624 20701 tablet.cc:2369] T 6ad2e80ac76d4833948d0b144469ffeb P a0404a3007e14b969155f38cdde5da50: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:10:06.143322 20724 raft_consensus.cc:482] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:10:06.143779 20724 raft_consensus.cc:504] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } }
I20260924 16:10:06.146211 20724 leader_election.cc:288] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0404a3007e14b969155f38cdde5da50 (127.3.214.195:42739), 4d2b7a5863b74f23b9c03656247e4c3a (127.3.214.194:40967)
I20260924 16:10:06.154274 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:06.146772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:45509 (local address 127.3.214.195:42739)
0924 16:10:06.147050 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:06.147056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:06.147713 (+ 657us) server_negotiation.cc:407] Connection header received
0924 16:10:06.148090 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:06.148098 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:06.148202 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:06.148347 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:10:06.149130 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:06.150045 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:06.152297 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:06.153177 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:06.153356 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:06.153449 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:06.153799 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 16:10:06.153947 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 16:10:06.154484 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:06.147982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:40995 (local address 127.3.214.194:40967)
0924 16:10:06.148237 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:06.148243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:06.148607 (+ 364us) server_negotiation.cc:407] Connection header received
0924 16:10:06.148778 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:06.148782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:06.148857 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:06.148940 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:10:06.149636 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:06.150582 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:06.152542 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:06.153401 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:06.153575 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:06.153667 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:06.153988 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 16:10:06.154175 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 16:10:06.154597 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:06.147752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:40967 (local address 127.3.214.193:40995)
0924 16:10:06.148461 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:06.148476 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:06.148630 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:06.148986 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:06.148991 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:06.149018 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:10:06.149443 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:06.149454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:06.150706 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:06.150717 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:06.152359 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:06.152373 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:06.153662 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:06.153668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:06.153817 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:06.153844 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 16:10:06.153941 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 16:10:06.154140 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":7,"thread_start_us":376,"threads_started":1}
I20260924 16:10:06.155092 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:06.146703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:42739 (local address 127.3.214.193:45509)
0924 16:10:06.147544 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:06.147566 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:06.147756 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:06.148452 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:06.148459 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:06.148505 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:10:06.148957 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:06.148970 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:06.150240 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:06.150247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:06.152083 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:06.152102 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:06.153426 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:06.153433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:06.153616 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:06.153649 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 16:10:06.153744 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 16:10:06.153922 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":35,"spinlock_wait_cycles":25984,"thread_start_us":447,"threads_started":1}
I20260924 16:10:06.156006 20520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad2e80ac76d4833948d0b144469ffeb" candidate_uuid: "32aa91a44d354548a6a62b87606e372f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" is_pre_election: true
I20260924 16:10:06.156481 20654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad2e80ac76d4833948d0b144469ffeb" candidate_uuid: "32aa91a44d354548a6a62b87606e372f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0404a3007e14b969155f38cdde5da50" is_pre_election: true
I20260924 16:10:06.156857 20520 raft_consensus.cc:2454] T 6ad2e80ac76d4833948d0b144469ffeb P 4d2b7a5863b74f23b9c03656247e4c3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32aa91a44d354548a6a62b87606e372f in term 0.
I20260924 16:10:06.157140 20654 raft_consensus.cc:2454] T 6ad2e80ac76d4833948d0b144469ffeb P a0404a3007e14b969155f38cdde5da50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32aa91a44d354548a6a62b87606e372f in term 0.
I20260924 16:10:06.157868 20320 leader_election.cc:302] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32aa91a44d354548a6a62b87606e372f, 4d2b7a5863b74f23b9c03656247e4c3a; no voters:
I20260924 16:10:06.158546 20724 raft_consensus.cc:2781] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:10:06.158763 20724 raft_consensus.cc:482] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:10:06.158877 20724 raft_consensus.cc:3037] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:10:06.161962 20724 raft_consensus.cc:504] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } }
I20260924 16:10:06.162931 20724 leader_election.cc:288] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 1 election: Requested vote from peers a0404a3007e14b969155f38cdde5da50 (127.3.214.195:42739), 4d2b7a5863b74f23b9c03656247e4c3a (127.3.214.194:40967)
W20260924 16:10:06.163053 20432 tablet.cc:2369] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:10:06.163645 20654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad2e80ac76d4833948d0b144469ffeb" candidate_uuid: "32aa91a44d354548a6a62b87606e372f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0404a3007e14b969155f38cdde5da50"
I20260924 16:10:06.163899 20654 raft_consensus.cc:3037] T 6ad2e80ac76d4833948d0b144469ffeb P a0404a3007e14b969155f38cdde5da50 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:10:06.164013 20520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad2e80ac76d4833948d0b144469ffeb" candidate_uuid: "32aa91a44d354548a6a62b87606e372f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d2b7a5863b74f23b9c03656247e4c3a"
I20260924 16:10:06.164343 20520 raft_consensus.cc:3037] T 6ad2e80ac76d4833948d0b144469ffeb P 4d2b7a5863b74f23b9c03656247e4c3a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:10:06.166653 20654 raft_consensus.cc:2454] T 6ad2e80ac76d4833948d0b144469ffeb P a0404a3007e14b969155f38cdde5da50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32aa91a44d354548a6a62b87606e372f in term 1.
I20260924 16:10:06.167238 20320 leader_election.cc:302] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32aa91a44d354548a6a62b87606e372f, a0404a3007e14b969155f38cdde5da50; no voters:
I20260924 16:10:06.167316 20520 raft_consensus.cc:2454] T 6ad2e80ac76d4833948d0b144469ffeb P 4d2b7a5863b74f23b9c03656247e4c3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32aa91a44d354548a6a62b87606e372f in term 1.
I20260924 16:10:06.167641 20724 raft_consensus.cc:2781] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:10:06.168313 20724 raft_consensus.cc:686] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 1 LEADER]: Becoming Leader. State: Replica: 32aa91a44d354548a6a62b87606e372f, State: Running, Role: LEADER
I20260924 16:10:06.168980 20724 consensus_queue.cc:237] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } }
I20260924 16:10:06.176033 20245 catalog_manager.cc:5495] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f reported cstate change: term changed from 0 to 1, leader changed from <none> to 32aa91a44d354548a6a62b87606e372f (127.3.214.193). New cstate: current_term: 1 leader_uuid: "32aa91a44d354548a6a62b87606e372f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } health_report { overall_health: UNKNOWN } } }
I20260924 16:10:06.209607 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:06.203424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46144 (local address 127.3.214.193:39785)
0924 16:10:06.203641 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:06.203647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:06.203669 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:10:06.203868 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:06.203872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:06.203977 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:06.204150 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:10:06.204583 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:06.205391 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:06.206984 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:06.207441 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:06.207551 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:06.208130 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:10:06.208918 (+ 788us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:10:06.209046 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:06.209165 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 16:10:06.209293 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 16:10:06.220167 20520 raft_consensus.cc:1261] T 6ad2e80ac76d4833948d0b144469ffeb P 4d2b7a5863b74f23b9c03656247e4c3a [term 1 FOLLOWER]: Refusing update from remote peer 32aa91a44d354548a6a62b87606e372f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:10:06.220206 20654 raft_consensus.cc:1261] T 6ad2e80ac76d4833948d0b144469ffeb P a0404a3007e14b969155f38cdde5da50 [term 1 FOLLOWER]: Refusing update from remote peer 32aa91a44d354548a6a62b87606e372f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:10:06.221387 20728 consensus_queue.cc:1035] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:10:06.221975 20724 consensus_queue.cc:1035] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:10:06.236356 20732 mvcc.cc:205] Tried to move back new op lower bound from 7332930380664836096 to 7332930380473573376. Current Snapshot: MvccSnapshot[applied={T|T < 7332930380664836096}]
I20260924 16:10:06.237478 20731 mvcc.cc:205] Tried to move back new op lower bound from 7332930380664836096 to 7332930380473573376. Current Snapshot: MvccSnapshot[applied={T|T < 7332930380664836096}]
I20260924 16:10:06.248153 20733 mvcc.cc:205] Tried to move back new op lower bound from 7332930380664836096 to 7332930380473573376. Current Snapshot: MvccSnapshot[applied={T|T < 7332930380664836096}]
I20260924 16:10:06.258296 20245 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:10:06.258641 20245 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:10:06.263171 20245 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7: Sending DeleteTablet for 3 replicas of tablet 6ad2e80ac76d4833948d0b144469ffeb
I20260924 16:10:06.265064 20366 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ad2e80ac76d4833948d0b144469ffeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:10:06 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:46134
I20260924 16:10:06.265270 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 20282
I20260924 16:10:06.265435 20498 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ad2e80ac76d4833948d0b144469ffeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:10:06 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:55282
I20260924 16:10:06.267649 20633 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ad2e80ac76d4833948d0b144469ffeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:10:06 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:55768
I20260924 16:10:06.280521 20743 tablet_replica.cc:330] T 6ad2e80ac76d4833948d0b144469ffeb P 4d2b7a5863b74f23b9c03656247e4c3a: stopping tablet replica
I20260924 16:10:06.282147 20743 raft_consensus.cc:2229] T 6ad2e80ac76d4833948d0b144469ffeb P 4d2b7a5863b74f23b9c03656247e4c3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:10:06.282907 20743 raft_consensus.cc:2258] T 6ad2e80ac76d4833948d0b144469ffeb P 4d2b7a5863b74f23b9c03656247e4c3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:10:06.288009 20743 ts_tablet_manager.cc:1905] T 6ad2e80ac76d4833948d0b144469ffeb P 4d2b7a5863b74f23b9c03656247e4c3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:10:06.303200 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:06.268684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46158 (local address 127.3.214.193:39785)
0924 16:10:06.268898 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:06.268905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:06.268930 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:10:06.269017 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:06.269027 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:06.269141 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:06.269296 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:06.272223 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:06.273925 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:06.276488 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:06.277410 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:06.277565 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:06.289815 (+ 12250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:06.289850 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:06.289884 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:06.289969 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:06.295150 (+ 5181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:06.296509 (+ 1359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:06.296521 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:06.296524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:06.296762 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:06.298831 (+ 2069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:06.298837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:06.298842 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:06.299554 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:06.299968 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:06.302059 (+ 2091us) server_negotiation.cc:299] Negotiation successful
0924 16:10:06.302738 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 16:10:06.304905 20744 tablet_replica.cc:330] T 6ad2e80ac76d4833948d0b144469ffeb P a0404a3007e14b969155f38cdde5da50: stopping tablet replica
I20260924 16:10:06.305199 20743 ts_tablet_manager.cc:1918] T 6ad2e80ac76d4833948d0b144469ffeb P 4d2b7a5863b74f23b9c03656247e4c3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:10:06.305413 20743 log.cc:1196] T 6ad2e80ac76d4833948d0b144469ffeb P 4d2b7a5863b74f23b9c03656247e4c3a: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/6ad2e80ac76d4833948d0b144469ffeb
I20260924 16:10:06.306329 20426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:10:06.307001 20744 raft_consensus.cc:2229] T 6ad2e80ac76d4833948d0b144469ffeb P a0404a3007e14b969155f38cdde5da50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:10:06.307266 20743 ts_tablet_manager.cc:1939] T 6ad2e80ac76d4833948d0b144469ffeb P 4d2b7a5863b74f23b9c03656247e4c3a: Deleting consensus metadata
I20260924 16:10:06.308051 20744 raft_consensus.cc:2258] T 6ad2e80ac76d4833948d0b144469ffeb P a0404a3007e14b969155f38cdde5da50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:10:06.310298 20230 catalog_manager.cc:4841] TS 4d2b7a5863b74f23b9c03656247e4c3a (127.3.214.194:40967): tablet 6ad2e80ac76d4833948d0b144469ffeb (table test-table [id=6e4392ca6c04441a89bb016cae319e8c]) successfully deleted
I20260924 16:10:06.312208 20744 ts_tablet_manager.cc:1905] T 6ad2e80ac76d4833948d0b144469ffeb P a0404a3007e14b969155f38cdde5da50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:10:06.318816 20744 ts_tablet_manager.cc:1918] T 6ad2e80ac76d4833948d0b144469ffeb P a0404a3007e14b969155f38cdde5da50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:10:06.318988 20744 log.cc:1196] T 6ad2e80ac76d4833948d0b144469ffeb P a0404a3007e14b969155f38cdde5da50: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/6ad2e80ac76d4833948d0b144469ffeb
I20260924 16:10:06.319434 20744 ts_tablet_manager.cc:1939] T 6ad2e80ac76d4833948d0b144469ffeb P a0404a3007e14b969155f38cdde5da50: Deleting consensus metadata
I20260924 16:10:06.321199 20230 catalog_manager.cc:4841] TS a0404a3007e14b969155f38cdde5da50 (127.3.214.195:42739): tablet 6ad2e80ac76d4833948d0b144469ffeb (table test-table [id=6e4392ca6c04441a89bb016cae319e8c]) successfully deleted
I20260924 16:10:06.425254 20742 tablet_replica.cc:330] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f: stopping tablet replica
I20260924 16:10:06.425436 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 20282
W20260924 16:10:06.445094 20232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.214.193:39785 (error 108)
I20260924 16:10:06.445760 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 20435
I20260924 16:10:06.447329 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:06.446743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:39785 (local address 127.0.0.1:46164)
0924 16:10:06.446978 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:06.447135 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:39785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260924 16:10:06.447664 20232 catalog_manager.cc:4584] TS 32aa91a44d354548a6a62b87606e372f (127.3.214.193:39785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6ad2e80ac76d4833948d0b144469ffeb: Network error: Client connection negotiation failed: client connection to 127.3.214.193:39785: connect: Connection refused (error 111)
I20260924 16:10:06.462777 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:06.446968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55292 (local address 127.3.214.194:40967)
0924 16:10:06.447193 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:06.447200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:06.447228 (+ 28us) server_negotiation.cc:407] Connection header received
0924 16:10:06.447434 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:06.447440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:06.447543 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:06.447664 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:06.449275 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:06.450349 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:06.451795 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:06.452223 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:06.452343 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:06.455784 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:06.455806 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:06.455827 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:06.455880 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:06.459400 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:06.460345 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:06.460354 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:06.460356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:06.460508 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:06.460892 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:06.460895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:06.460897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:06.461337 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:06.461665 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:06.462002 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 16:10:06.462412 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 16:10:06.463841 20560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:10:06.501835 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:06.500893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:39785 (local address 127.0.0.1:46176)
0924 16:10:06.501275 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:06.501457 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:39785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260924 16:10:06.503463 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:06.502955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:39785 (local address 127.0.0.1:46192)
0924 16:10:06.503173 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:06.503303 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:39785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 16:10:06.565644 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:06.564877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:39785 (local address 127.0.0.1:46198)
0924 16:10:06.565265 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:06.565394 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:39785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215}
I20260924 16:10:06.565912 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 20435
I20260924 16:10:06.569039 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:06.568405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:39785 (local address 127.0.0.1:46200)
0924 16:10:06.568739 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:06.568859 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:39785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260924 16:10:06.586915 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 20569
I20260924 16:10:06.611855 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:06.588030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.3.214.195:42739)
0924 16:10:06.588242 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:06.588246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:06.588270 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:10:06.588454 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:06.588458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:06.588545 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:06.588665 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:06.590336 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:06.591546 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:06.593550 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:06.594355 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:06.594517 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:06.598427 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:06.598454 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:06.598482 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:06.598538 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:06.608302 (+ 9764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:06.609364 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:06.609377 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:06.609381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:06.609556 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:06.609966 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:06.609972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:06.609975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:06.610510 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:06.610828 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:06.611180 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 16:10:06.611448 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 16:10:06.612766 20695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:10:06.650380 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:06.649629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:39785 (local address 127.0.0.1:46212)
0924 16:10:06.649998 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:06.650167 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:39785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260924 16:10:06.651579 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:06.651164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:39785 (local address 127.0.0.1:46214)
0924 16:10:06.651376 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:06.651455 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:39785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260924 16:10:06.714093 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 20569
I20260924 16:10:06.734673 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 20212
I20260924 16:10:06.736384 20275 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:10:06.858249 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:06.856903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:39785 (local address 127.0.0.1:46226)
0924 16:10:06.857639 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:06.857811 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:39785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563}
I20260924 16:10:06.858686 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 20212
I20260924 16:10:26.883900 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:40829
--webserver_interface=127.3.214.254
--webserver_port=38721
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:40829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:10:27.294732 20750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:10:27.295208 20750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:10:27.295382 20750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:10:27.303907 20750 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 16:10:27.304070 20750 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 16:10:27.304113 20750 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 16:10:27.304163 20750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:10:27.304234 20750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:10:27.304337 20750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:10:27.304409 20750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:10:27.304490 20750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:10:27.316478 20750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:40829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:40829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=38721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:10:27.318419 20750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:10:27.320240 20750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:10:27.335314 20758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:27.335462 20755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:27.335408 20756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:27.336285 20750 server_base.cc:1034] running on GCE node
I20260924 16:10:27.337320 20750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:10:27.339182 20750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:10:27.340507 20750 hybrid_clock.cc:648] HybridClock initialized: now 1790266227340393 us; error 76 us; skew 500 ppm
I20260924 16:10:27.340956 20750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:10:27 dist-test-slave-1gc3 krb5kdc[19656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266227, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:10:27.348968 20750 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:10:27.351148 20750 webserver.cc:460] Webserver started at http://127.3.214.254:38721/ using document root <none> and password file <none>
I20260924 16:10:27.352072 20750 fs_manager.cc:359] Metadata directory not provided
I20260924 16:10:27.352214 20750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:10:27.360414 20750 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 16:10:27.364710 20765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:27.366811 20750 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:10:27.367046 20750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "2ae01576d2fc4609972bb444423d12a7"
format_stamp: "Formatted at 2026-09-24 16:09:40 on dist-test-slave-1gc3"
server_key: "75d417bd5c0bf24aac1c7a7c34763393"
server_key_iv: "72257ba305e7e54adba06fc624aad30e"
server_key_version: "encryptionkey@0"
I20260924 16:10:27.368227 20750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:10:27.387882 20750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:10:27.392956 20750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:10:27.393319 20750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:10:27.415346 20750 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:40829
I20260924 16:10:27.415417 20817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:40829 every 8 connection(s)
I20260924 16:10:27.417174 20750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:10:27.426360 20818 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:10:27.427589 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 20750
I20260924 16:10:27.428501 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:39785
--local_ip_for_outbound_sockets=127.3.214.193
--tserver_master_addrs=127.3.214.254:40829
--webserver_port=43921
--webserver_interface=127.3.214.193
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:10:27.430693 20818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7: Bootstrap starting.
I20260924 16:10:27.456861 20818 log.cc:824] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7: Log is configured to *not* fsync() on all Append() calls
I20260924 16:10:27.476082 20818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:10:27.476758 20818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7: Bootstrap complete.
I20260924 16:10:27.487154 20818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae01576d2fc4609972bb444423d12a7" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40829 } }
I20260924 16:10:27.488281 20818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae01576d2fc4609972bb444423d12a7, State: Initialized, Role: FOLLOWER
I20260924 16:10:27.489105 20818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae01576d2fc4609972bb444423d12a7" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40829 } }
I20260924 16:10:27.489389 20818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:10:27.489519 20818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:10:27.489705 20818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 2 FOLLOWER]: Advancing to term 3
I20260924 16:10:27.494521 20818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae01576d2fc4609972bb444423d12a7" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40829 } }
I20260924 16:10:27.495178 20818 leader_election.cc:302] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ae01576d2fc4609972bb444423d12a7; no voters:
I20260924 16:10:27.495975 20818 leader_election.cc:288] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 16:10:27.496136 20822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 16:10:27.497586 20822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [term 3 LEADER]: Becoming Leader. State: Replica: 2ae01576d2fc4609972bb444423d12a7, State: Running, Role: LEADER
I20260924 16:10:27.498482 20822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae01576d2fc4609972bb444423d12a7" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40829 } }
I20260924 16:10:27.499076 20818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:10:27.501653 20824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ae01576d2fc4609972bb444423d12a7. Latest consensus state: current_term: 3 leader_uuid: "2ae01576d2fc4609972bb444423d12a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae01576d2fc4609972bb444423d12a7" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40829 } } }
I20260924 16:10:27.502275 20824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:10:27.502560 20823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2ae01576d2fc4609972bb444423d12a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae01576d2fc4609972bb444423d12a7" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40829 } } }
I20260924 16:10:27.502976 20823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:10:27.513257 20829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:10:27.524112 20829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:10:27.526192 20829 catalog_manager.cc:1259] Loaded cluster ID: e809d8a952f6463ab15ca057e2e792a5
I20260924 16:10:27.526433 20829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:10:27.531080 20829 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:10:27.533071 20829 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7: Loaded TSK: 0
I20260924 16:10:27.533751 20829 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2ae01576d2fc4609972bb444423d12a7: Loaded TSK: 1
F20260924 16:10:27.535137 20829 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f871f1b79ad google::LogMessage::Fail() at ??:0
@ 0x7f871f1ba88a google::LogMessage::SendToLog() at ??:0
@ 0x7f871f1b74de google::LogMessage::Flush() at ??:0
@ 0x7f871f1b8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8722f1b7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f8722f0a73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f8734be61c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f8734be5783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f8734c712a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f8734c710c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f8734cbb5e9 std::function<>::operator()() at ??:0
@ 0x7f8734be9cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f8734be7c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8734c6f610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f8734c6f439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f8721e03345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8721e0691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f8721e06799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f8721de1eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f87250bb6db start_thread at ??:0
@ 0x7f871e05161f clone at ??:0
W20260924 16:10:27.858018 20820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:10:27.858384 20820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:10:27.858521 20820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:10:27.866287 20820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:10:27.866422 20820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:10:27.866583 20820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:10:27.876888 20820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:39785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=43921
--tserver_master_addrs=127.3.214.254:40829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:10:27.878378 20820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:10:27.879782 20820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:10:27.894145 20866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:27.894266 20865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:27.894484 20868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:27.895632 20820 server_base.cc:1034] running on GCE node
I20260924 16:10:27.896361 20820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:10:27.897564 20820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:10:27.898810 20820 hybrid_clock.cc:648] HybridClock initialized: now 1790266227898728 us; error 71 us; skew 500 ppm
I20260924 16:10:27.899220 20820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:10:27 dist-test-slave-1gc3 krb5kdc[19656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266227, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:10:27.902848 20820 init.cc:377] Logged in from keytab as kudu/127.3.214.193@KRBTEST.COM (short username kudu)
I20260924 16:10:27.904807 20820 webserver.cc:460] Webserver started at http://127.3.214.193:43921/ using document root <none> and password file <none>
I20260924 16:10:27.905758 20820 fs_manager.cc:359] Metadata directory not provided
I20260924 16:10:27.905901 20820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:10:27.912804 20820 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.001s
I20260924 16:10:27.916780 20875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:27.918797 20820 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:10:27.919050 20820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "32aa91a44d354548a6a62b87606e372f"
format_stamp: "Formatted at 2026-09-24 16:09:41 on dist-test-slave-1gc3"
server_key: "9c78082edc6a91d8166f6c3e36eceab8"
server_key_iv: "cd71c2897678fbb9056137e0c837911e"
server_key_version: "encryptionkey@0"
I20260924 16:10:27.920356 20820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:10:27.949958 20820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:10:27.955729 20820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:10:27.956068 20820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:10:27.957541 20820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:10:27.960003 20883 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 16:10:27.966712 20820 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 16:10:27.966955 20820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260924 16:10:27.967159 20820 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 16:10:27.971504 20820 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 16:10:27.971634 20820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.004s sys 0.000s
I20260924 16:10:27.972224 20883 tablet_bootstrap.cc:492] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f: Bootstrap starting.
I20260924 16:10:28.000954 20883 log.cc:824] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:10:28.016527 20820 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:39785
I20260924 16:10:28.016546 20991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:39785 every 8 connection(s)
I20260924 16:10:28.016880 20883 tablet_bootstrap.cc:492] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:10:28.017494 20883 tablet_bootstrap.cc:492] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f: Bootstrap complete.
I20260924 16:10:28.018397 20820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:10:28.018911 20883 ts_tablet_manager.cc:1397] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f: Time spent bootstrapping tablet: real 0.047s user 0.028s sys 0.017s
I20260924 16:10:28.025482 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:28.022431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.193:35131)
0924 16:10:28.023689 (+ 1258us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:28.024030 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":845,"thread_start_us":527,"threads_started":1}
W20260924 16:10:28.026403 20992 heartbeater.cc:644] Failed to heartbeat to 127.3.214.254:40829 (0 consecutive failures): Network error: Failed to ping master at 127.3.214.254:40829: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
I20260924 16:10:28.027786 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:28.027210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.193:59975)
0924 16:10:28.027458 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:28.027608 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 16:10:28.028920 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 20820
I20260924 16:10:28.029724 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:40967
--local_ip_for_outbound_sockets=127.3.214.194
--tserver_master_addrs=127.3.214.254:40829
--webserver_port=45585
--webserver_interface=127.3.214.194
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:10:28.028573 20883 raft_consensus.cc:348] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } }
I20260924 16:10:28.030035 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:28.029497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.193:57101)
0924 16:10:28.029691 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:28.029815 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 16:10:28.030162 20883 raft_consensus.cc:729] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32aa91a44d354548a6a62b87606e372f, State: Initialized, Role: FOLLOWER
W20260924 16:10:28.030406 20992 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 16:10:28.031198 20883 consensus_queue.cc:260] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } }
I20260924 16:10:28.033720 20883 ts_tablet_manager.cc:1428] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260924 16:10:28.035317 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:28.034048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.193:50979)
0924 16:10:28.035005 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:28.035125 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":799}
I20260924 16:10:28.036394 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:28.036002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.193:50985)
0924 16:10:28.036160 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:28.036243 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 16:10:28.390414 20996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:10:28.390758 20996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:10:28.390892 20996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:10:28.398586 20996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:10:28.398742 20996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:10:28.398912 20996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:10:28.410458 20996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:40967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=45585
--tserver_master_addrs=127.3.214.254:40829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:10:28.411937 20996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:10:28.413365 20996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:10:28.429407 21002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:28.429493 21001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:28.429513 21004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:28.430590 20996 server_base.cc:1034] running on GCE node
I20260924 16:10:28.431442 20996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:10:28.432750 20996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:10:28.434012 20996 hybrid_clock.cc:648] HybridClock initialized: now 1790266228433941 us; error 51 us; skew 500 ppm
I20260924 16:10:28.434468 20996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:10:28 dist-test-slave-1gc3 krb5kdc[19656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266228, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:10:28.438689 20996 init.cc:377] Logged in from keytab as kudu/127.3.214.194@KRBTEST.COM (short username kudu)
I20260924 16:10:28.440655 20996 webserver.cc:460] Webserver started at http://127.3.214.194:45585/ using document root <none> and password file <none>
I20260924 16:10:28.441565 20996 fs_manager.cc:359] Metadata directory not provided
I20260924 16:10:28.441684 20996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:10:28.448848 20996 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:10:28.452740 21011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:28.454807 20996 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:10:28.455052 20996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "4d2b7a5863b74f23b9c03656247e4c3a"
format_stamp: "Formatted at 2026-09-24 16:09:41 on dist-test-slave-1gc3"
server_key: "efdacfceba9f987c930af1dbdfc25b85"
server_key_iv: "cecbc330ff8ac6164e4a2812d8ffda9a"
server_key_version: "encryptionkey@0"
I20260924 16:10:28.456336 20996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:10:28.476397 20996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:10:28.481238 20996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:10:28.481552 20996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:10:28.482833 20996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:10:28.484536 20996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:10:28.484638 20996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:28.484757 20996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:10:28.484830 20996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:28.522816 20996 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:40967
I20260924 16:10:28.522902 21125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:40967 every 8 connection(s)
I20260924 16:10:28.524623 20996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:10:28.530619 21128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:28.528174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.194:35875)
0924 16:10:28.529102 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:28.529439 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":339,"threads_started":1}
W20260924 16:10:28.531538 21126 heartbeater.cc:644] Failed to heartbeat to 127.3.214.254:40829 (0 consecutive failures): Network error: Failed to ping master at 127.3.214.254:40829: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
I20260924 16:10:28.532668 21128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:28.532238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.194:42607)
0924 16:10:28.532446 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:28.532537 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 16:10:28.532668 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 20996
I20260924 16:10:28.533291 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:42739
--local_ip_for_outbound_sockets=127.3.214.195
--tserver_master_addrs=127.3.214.254:40829
--webserver_port=41353
--webserver_interface=127.3.214.195
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:10:28.533860 21128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:28.533391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.194:35057)
0924 16:10:28.533629 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:28.533712 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260924 16:10:28.534315 21126 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:10:28.941543 21129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:10:28.941965 21129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:10:28.942130 21129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:10:28.950670 21129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:10:28.950850 21129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:10:28.951028 21129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:10:28.962558 21129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:42739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=41353
--tserver_master_addrs=127.3.214.254:40829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:10:28.964432 21129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:10:28.966197 21129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:10:28.982291 21134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:28.982399 21137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:28.982388 21135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:28.983403 21129 server_base.cc:1034] running on GCE node
I20260924 16:10:28.984251 21129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:10:28.985347 21129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:10:28.986621 21129 hybrid_clock.cc:648] HybridClock initialized: now 1790266228986530 us; error 68 us; skew 500 ppm
I20260924 16:10:28.987073 21129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:10:28 dist-test-slave-1gc3 krb5kdc[19656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266228, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:10:28.990701 21129 init.cc:377] Logged in from keytab as kudu/127.3.214.195@KRBTEST.COM (short username kudu)
I20260924 16:10:28.992624 21129 webserver.cc:460] Webserver started at http://127.3.214.195:41353/ using document root <none> and password file <none>
I20260924 16:10:28.993512 21129 fs_manager.cc:359] Metadata directory not provided
I20260924 16:10:28.993651 21129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:10:29.001072 21129 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 16:10:29.005007 21144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:29.007054 21129 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:10:29.007295 21129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "a0404a3007e14b969155f38cdde5da50"
format_stamp: "Formatted at 2026-09-24 16:09:42 on dist-test-slave-1gc3"
server_key: "ed460bb15d3791f3fed5f64e0c007bb2"
server_key_iv: "223e26642e659b0bf4f3473ad35d25dc"
server_key_version: "encryptionkey@0"
I20260924 16:10:29.008460 21129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:10:29.034154 21129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:10:29.038867 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:29.037568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.193:56811)
0924 16:10:29.038448 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:29.038565 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":550,"threads_started":1}
I20260924 16:10:29.040819 21129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:10:29.041086 21129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:10:29.042289 21129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:10:29.043917 21129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:10:29.044027 21129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:29.044130 21129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:10:29.044190 21129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:29.079859 21129 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:42739
I20260924 16:10:29.079965 21258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:42739 every 8 connection(s)
I20260924 16:10:29.081555 21129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:10:29.086468 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 21129
I20260924 16:10:29.088680 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:29.085620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.195:60201)
0924 16:10:29.086834 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:29.087254 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":480,"threads_started":1}
W20260924 16:10:29.089614 21259 heartbeater.cc:644] Failed to heartbeat to 127.3.214.254:40829 (0 consecutive failures): Network error: Failed to ping master at 127.3.214.254:40829: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
I20260924 16:10:29.090655 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:29.090284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.195:34015)
0924 16:10:29.090445 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:29.090526 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 16:10:29.091588 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:29.091227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.195:60259)
0924 16:10:29.091372 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:29.091450 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 16:10:29.091846 21259 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 16:10:29.330196 21264 raft_consensus.cc:482] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:10:29.330535 21264 raft_consensus.cc:504] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } }
I20260924 16:10:29.332347 21264 leader_election.cc:288] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0404a3007e14b969155f38cdde5da50 (127.3.214.195:42739), 4d2b7a5863b74f23b9c03656247e4c3a (127.3.214.194:40967)
Sep 24 16:10:29 dist-test-slave-1gc3 krb5kdc[19656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266227, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.195@KRBTEST.COM
Sep 24 16:10:29 dist-test-slave-1gc3 krb5kdc[19656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266227, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.194@KRBTEST.COM
I20260924 16:10:29.357652 21266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:29.333539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:40967 (local address 127.3.214.193:46565)
0924 16:10:29.334699 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:29.334722 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:29.334893 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:29.337659 (+ 2766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:29.337664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:29.338502 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:10:29.339403 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:29.339444 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:29.341251 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:29.341256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:29.343264 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:29.343281 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:29.344610 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:29.344617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:29.344937 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:29.347500 (+ 2563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:10:29.347520 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:10:29.349521 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:10:29.353311 (+ 3790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:10:29.353318 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:10:29.353323 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:10:29.353889 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:10:29.354485 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:10:29.354491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:10:29.354495 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:10:29.354698 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:10:29.355648 (+ 950us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:10:29.355652 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:10:29.356020 (+ 368us) client_negotiation.cc:769] Sending connection context
0924 16:10:29.356365 (+ 345us) client_negotiation.cc:240] Negotiation successful
0924 16:10:29.357130 (+ 765us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"mutex_wait_us":264,"spinlock_wait_cycles":9472,"thread_start_us":857,"threads_started":1}
I20260924 16:10:29.358160 21265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:29.333191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:56181 (local address 127.3.214.195:42739)
0924 16:10:29.333965 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:29.333984 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:29.334044 (+ 60us) server_negotiation.cc:407] Connection header received
0924 16:10:29.336374 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:29.336384 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:29.336870 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:29.337241 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:29.339684 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:29.341019 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:29.343493 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:29.344262 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:29.344993 (+ 731us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:29.349472 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:29.349483 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:29.349505 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:29.349568 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:29.352852 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:29.354035 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:29.354045 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:29.354051 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:29.354304 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:29.354827 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:29.354831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:29.354832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:29.355447 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:29.355660 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:29.357068 (+ 1408us) server_negotiation.cc:299] Negotiation successful
0924 16:10:29.357665 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":352,"threads_started":1}
I20260924 16:10:29.358547 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:29.332954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:42739 (local address 127.3.214.193:56181)
0924 16:10:29.333209 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:29.333277 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:29.334125 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:29.337496 (+ 3371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:29.337508 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:29.338483 (+ 975us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:10:29.339408 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:29.339441 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:29.341268 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:29.341272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:29.343268 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:29.343291 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:29.344576 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:29.344584 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:29.344950 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:29.345994 (+ 1044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:10:29.346035 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:10:29.348989 (+ 2954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:10:29.353084 (+ 4095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:10:29.353095 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:10:29.353117 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:10:29.353854 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:10:29.354434 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:10:29.354440 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:10:29.354444 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:10:29.354694 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:10:29.355607 (+ 913us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:10:29.355617 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:10:29.356033 (+ 416us) client_negotiation.cc:769] Sending connection context
0924 16:10:29.356364 (+ 331us) client_negotiation.cc:240] Negotiation successful
0924 16:10:29.357725 (+ 1361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":940,"spinlock_wait_cycles":29184}
I20260924 16:10:29.358596 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:29.334378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:46565 (local address 127.3.214.194:40967)
0924 16:10:29.335106 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:29.335119 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:29.335157 (+ 38us) server_negotiation.cc:407] Connection header received
0924 16:10:29.336834 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:29.336846 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:29.337313 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:29.337656 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:29.339663 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:29.341008 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:29.343497 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:29.344315 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:29.344528 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:29.349861 (+ 5333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:29.349872 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:29.349889 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:29.349945 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:29.353141 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:29.354065 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:29.354074 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:29.354083 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:29.354353 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:29.354841 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:29.354845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:29.354847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:29.355514 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:29.355739 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:29.357364 (+ 1625us) server_negotiation.cc:299] Negotiation successful
0924 16:10:29.358146 (+ 782us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":395,"threads_started":1}
W20260924 16:10:29.362646 20877 leader_election.cc:341] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0404a3007e14b969155f38cdde5da50 (127.3.214.195:42739): Not found: Tablet not found: 6ad2e80ac76d4833948d0b144469ffeb
W20260924 16:10:29.363235 20877 leader_election.cc:341] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4d2b7a5863b74f23b9c03656247e4c3a (127.3.214.194:40967): Not found: Tablet not found: 6ad2e80ac76d4833948d0b144469ffeb
I20260924 16:10:29.363420 20877 leader_election.cc:302] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32aa91a44d354548a6a62b87606e372f; no voters: 4d2b7a5863b74f23b9c03656247e4c3a, a0404a3007e14b969155f38cdde5da50
I20260924 16:10:29.363987 21264 raft_consensus.cc:2726] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:10:29.536195 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:29.535056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.194:38733)
0924 16:10:29.535719 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:29.535898 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":323,"threads_started":1}
I20260924 16:10:30.041452 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:30.040255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.193:51173)
0924 16:10:30.041032 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:30.041175 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":414,"threads_started":1}
I20260924 16:10:30.093597 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:30.092612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.195:50459)
0924 16:10:30.093167 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:30.093353 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":262,"threads_started":1}
I20260924 16:10:30.538491 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:30.537346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.194:43459)
0924 16:10:30.538141 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:30.538277 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":393,"threads_started":1}
I20260924 16:10:31.044184 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:31.042891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.193:49967)
0924 16:10:31.043723 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:31.043877 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":347,"threads_started":1}
I20260924 16:10:31.096096 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:31.095034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.195:40945)
0924 16:10:31.095733 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:31.095860 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":375,"threads_started":1}
I20260924 16:10:31.167820 21274 raft_consensus.cc:482] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:10:31.168109 21274 raft_consensus.cc:504] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } }
I20260924 16:10:31.168905 21274 leader_election.cc:288] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0404a3007e14b969155f38cdde5da50 (127.3.214.195:42739), 4d2b7a5863b74f23b9c03656247e4c3a (127.3.214.194:40967)
W20260924 16:10:31.170145 20877 leader_election.cc:341] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0404a3007e14b969155f38cdde5da50 (127.3.214.195:42739): Not found: Tablet not found: 6ad2e80ac76d4833948d0b144469ffeb
W20260924 16:10:31.170444 20877 leader_election.cc:341] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4d2b7a5863b74f23b9c03656247e4c3a (127.3.214.194:40967): Not found: Tablet not found: 6ad2e80ac76d4833948d0b144469ffeb
I20260924 16:10:31.170542 20877 leader_election.cc:302] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32aa91a44d354548a6a62b87606e372f; no voters: 4d2b7a5863b74f23b9c03656247e4c3a, a0404a3007e14b969155f38cdde5da50
I20260924 16:10:31.170888 21274 raft_consensus.cc:2726] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:10:31.540946 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:31.539879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.194:57817)
0924 16:10:31.540581 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:31.540717 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":372,"threads_started":1}
I20260924 16:10:32.046814 21276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:32.045690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.193:36271)
0924 16:10:32.046425 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:32.046567 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":417,"threads_started":1}
I20260924 16:10:32.098619 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:32.097496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.195:50151)
0924 16:10:32.098238 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:32.098378 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":423,"threads_started":1}
I20260924 16:10:32.543150 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:32.542194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.194:42395)
0924 16:10:32.542816 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:32.542925 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":315,"threads_started":1}
I20260924 16:10:32.823470 21279 raft_consensus.cc:482] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:10:32.823733 21279 raft_consensus.cc:504] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } }
I20260924 16:10:32.824455 21279 leader_election.cc:288] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0404a3007e14b969155f38cdde5da50 (127.3.214.195:42739), 4d2b7a5863b74f23b9c03656247e4c3a (127.3.214.194:40967)
W20260924 16:10:32.825799 20877 leader_election.cc:341] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4d2b7a5863b74f23b9c03656247e4c3a (127.3.214.194:40967): Not found: Tablet not found: 6ad2e80ac76d4833948d0b144469ffeb
W20260924 16:10:32.826067 20877 leader_election.cc:341] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0404a3007e14b969155f38cdde5da50 (127.3.214.195:42739): Not found: Tablet not found: 6ad2e80ac76d4833948d0b144469ffeb
I20260924 16:10:32.826224 20877 leader_election.cc:302] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32aa91a44d354548a6a62b87606e372f; no voters: 4d2b7a5863b74f23b9c03656247e4c3a, a0404a3007e14b969155f38cdde5da50
I20260924 16:10:32.826531 21279 raft_consensus.cc:2726] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:10:33.049410 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:33.048417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.193:54219)
0924 16:10:33.049055 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:33.049177 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":308,"threads_started":1}
I20260924 16:10:33.101075 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:33.100012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.195:56037)
0924 16:10:33.100689 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:33.100813 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":360,"threads_started":1}
I20260924 16:10:33.545872 21282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:33.544551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.194:42855)
0924 16:10:33.545490 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:33.545621 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":360,"threads_started":1}
I20260924 16:10:34.051594 21283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:34.050608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.193:39571)
0924 16:10:34.051250 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:34.051392 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":324,"threads_started":1}
I20260924 16:10:34.103587 21284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:34.102593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.195:60013)
0924 16:10:34.103218 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:34.103356 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":331,"threads_started":1}
I20260924 16:10:34.548372 21285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:34.547375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.194:44215)
0924 16:10:34.548020 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:34.548155 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":335,"threads_started":1}
I20260924 16:10:35.053716 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:35.052796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.193:53803)
0924 16:10:35.053386 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:35.053500 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":311,"threads_started":1}
I20260924 16:10:35.106053 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:35.104989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.195:54363)
0924 16:10:35.105641 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:35.105776 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":322,"threads_started":1}
I20260924 16:10:35.550849 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:35.549604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.194:43557)
0924 16:10:35.550461 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:35.550595 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":336,"threads_started":1}
I20260924 16:10:36.055932 21289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:36.054864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.193:51241)
0924 16:10:36.055594 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:36.055715 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":423,"threads_started":1}
I20260924 16:10:36.108668 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:36.107528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.195:37219)
0924 16:10:36.108299 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:36.108429 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":341,"threads_started":1}
I20260924 16:10:36.553257 21291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:36.552250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.194:54013)
0924 16:10:36.552893 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:36.553011 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":336,"threads_started":1}
I20260924 16:10:37.058243 21292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:37.057252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.193:40341)
0924 16:10:37.057852 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:37.057965 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":292,"threads_started":1}
I20260924 16:10:37.111162 21293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:37.110142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.195:33539)
0924 16:10:37.110808 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:37.110924 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":332,"threads_started":1}
I20260924 16:10:37.464582 21294 raft_consensus.cc:482] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:10:37.464823 21294 raft_consensus.cc:504] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } }
I20260924 16:10:37.465530 21294 leader_election.cc:288] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0404a3007e14b969155f38cdde5da50 (127.3.214.195:42739), 4d2b7a5863b74f23b9c03656247e4c3a (127.3.214.194:40967)
W20260924 16:10:37.466760 20877 leader_election.cc:341] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0404a3007e14b969155f38cdde5da50 (127.3.214.195:42739): Not found: Tablet not found: 6ad2e80ac76d4833948d0b144469ffeb
W20260924 16:10:37.467013 20877 leader_election.cc:341] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4d2b7a5863b74f23b9c03656247e4c3a (127.3.214.194:40967): Not found: Tablet not found: 6ad2e80ac76d4833948d0b144469ffeb
I20260924 16:10:37.467103 20877 leader_election.cc:302] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32aa91a44d354548a6a62b87606e372f; no voters: 4d2b7a5863b74f23b9c03656247e4c3a, a0404a3007e14b969155f38cdde5da50
I20260924 16:10:37.467337 21294 raft_consensus.cc:2726] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:10:37.555593 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:37.554555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.194:38663)
0924 16:10:37.555299 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:37.555399 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":427,"threads_started":1}
I20260924 16:10:38.060899 21296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:38.059803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.193:44273)
0924 16:10:38.060538 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:38.060667 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":341,"threads_started":1}
I20260924 16:10:38.113546 21297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:38.112555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.195:47893)
0924 16:10:38.113241 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:38.113363 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":370,"threads_started":1}
I20260924 16:10:38.558043 21298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:38.557018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.194:36263)
0924 16:10:38.557652 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:38.557806 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":335,"threads_started":1}
I20260924 16:10:39.063385 21299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:39.062258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.193:45771)
0924 16:10:39.062929 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:39.063112 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":322,"threads_started":1}
I20260924 16:10:39.115720 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:39.114788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.195:35431)
0924 16:10:39.115379 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:39.115496 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":309,"threads_started":1}
I20260924 16:10:39.560653 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:39.559549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.194:48557)
0924 16:10:39.560268 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:39.560430 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":351,"threads_started":1}
I20260924 16:10:40.065318 21302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:40.064417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.193:60937)
0924 16:10:40.064996 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:40.065101 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":331,"threads_started":1}
I20260924 16:10:40.117985 21303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:40.116938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.195:50681)
0924 16:10:40.117665 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:40.117789 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":354,"threads_started":1}
I20260924 16:10:40.563184 21304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:40.562080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.194:52755)
0924 16:10:40.562775 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:40.562944 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":310,"threads_started":1}
I20260924 16:10:41.067693 21305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:41.066663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.193:44915)
0924 16:10:41.067306 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:41.067441 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":321,"threads_started":1}
I20260924 16:10:41.120316 21306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:41.119381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.195:48899)
0924 16:10:41.120007 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:41.120132 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":313,"threads_started":1}
I20260924 16:10:41.565500 21307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:41.564385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.194:39355)
0924 16:10:41.565042 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:41.565292 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":308,"threads_started":1}
I20260924 16:10:42.069881 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:42.068999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.193:58755)
0924 16:10:42.069534 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:42.069635 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":290,"threads_started":1}
I20260924 16:10:42.122591 21309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:42.121587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.195:40421)
0924 16:10:42.122260 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:42.122411 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":335,"threads_started":1}
I20260924 16:10:42.567966 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:42.566917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.194:41379)
0924 16:10:42.567563 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:42.567703 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":309,"threads_started":1}
I20260924 16:10:43.072638 21311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:43.071674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.193:55439)
0924 16:10:43.072305 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:43.072432 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":332,"threads_started":1}
I20260924 16:10:43.124689 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:43.123750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.195:35311)
0924 16:10:43.124375 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:43.124478 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":347,"threads_started":1}
I20260924 16:10:43.570603 21313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:43.569566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.194:55897)
0924 16:10:43.570205 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:43.570384 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":284,"threads_started":1}
I20260924 16:10:44.074935 21314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:44.073939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.193:53761)
0924 16:10:44.074588 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:44.074712 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":342,"threads_started":1}
I20260924 16:10:44.099083 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 20820
I20260924 16:10:44.127305 21316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:44.126203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.195:34741)
0924 16:10:44.126944 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:44.127122 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":333,"threads_started":1}
I20260924 16:10:44.357553 21317 raft_consensus.cc:482] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:10:44.357803 21317 raft_consensus.cc:504] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0404a3007e14b969155f38cdde5da50" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 42739 } } peers { permanent_uuid: "32aa91a44d354548a6a62b87606e372f" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 39785 } } peers { permanent_uuid: "4d2b7a5863b74f23b9c03656247e4c3a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 40967 } }
I20260924 16:10:44.358994 21317 leader_election.cc:288] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0404a3007e14b969155f38cdde5da50 (127.3.214.195:42739), 4d2b7a5863b74f23b9c03656247e4c3a (127.3.214.194:40967)
W20260924 16:10:44.360646 20877 leader_election.cc:341] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0404a3007e14b969155f38cdde5da50 (127.3.214.195:42739): Not found: Tablet not found: 6ad2e80ac76d4833948d0b144469ffeb
W20260924 16:10:44.360961 20877 leader_election.cc:341] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4d2b7a5863b74f23b9c03656247e4c3a (127.3.214.194:40967): Not found: Tablet not found: 6ad2e80ac76d4833948d0b144469ffeb
I20260924 16:10:44.361096 20877 leader_election.cc:302] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32aa91a44d354548a6a62b87606e372f; no voters: 4d2b7a5863b74f23b9c03656247e4c3a, a0404a3007e14b969155f38cdde5da50
I20260924 16:10:44.361521 21317 raft_consensus.cc:2726] T 6ad2e80ac76d4833948d0b144469ffeb P 32aa91a44d354548a6a62b87606e372f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:10:44.573027 21318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:44.572017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40829 (local address 127.3.214.194:44699)
0924 16:10:44.572647 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:44.572799 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.254:40829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":301,"threads_started":1}
I20260924 16:10:44.752190 21315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:44.101311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.3.214.193:39785)
0924 16:10:44.102180 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:44.102192 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:44.730972 (+628780us) server_negotiation.cc:407] Connection header received
0924 16:10:44.731168 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:44.731178 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:44.731540 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:44.731744 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:44.733717 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:44.734900 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:44.736585 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:44.737266 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:44.737392 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:44.744591 (+ 7199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:44.744616 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:44.744641 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:44.744711 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:44.748183 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:44.749350 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:44.749361 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:44.749364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:44.749534 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:44.750143 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:44.750150 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:44.750154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:44.750705 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:44.751038 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:44.751442 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 16:10:44.751718 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":330,"threads_started":1}
I20260924 16:10:44.753922 20987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:10:44.851320 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 20820
I20260924 16:10:44.873459 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 20996
I20260924 16:10:44.890736 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:44.874475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46290 (local address 127.3.214.194:40967)
0924 16:10:44.875180 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:44.875186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:44.875209 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:10:44.875286 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:44.875290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:44.875366 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:44.875481 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:44.877195 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:44.878241 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:44.879703 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:44.880220 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:44.880357 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:44.883565 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:44.883572 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:44.883577 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:44.883651 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:44.887054 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:44.888009 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:44.888015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:44.888017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:44.888237 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:44.888737 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:44.888743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:44.888746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:44.889152 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:44.889468 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:44.889913 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 16:10:44.890336 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":380,"threads_started":1}
I20260924 16:10:44.891721 21121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:10:44.990157 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 20996
I20260924 16:10:45.009645 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 21129
I20260924 16:10:45.027122 21322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:45.010666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46516 (local address 127.3.214.195:42739)
0924 16:10:45.011384 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:45.011390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:45.011413 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:10:45.011488 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:45.011492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:45.011570 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:45.011682 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:45.013289 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:45.014435 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:45.015923 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:45.016434 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:45.016568 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:45.019595 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:45.019602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:45.019607 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:45.019666 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:45.023192 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:45.024217 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:45.024223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:45.024247 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:45.024490 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:45.025094 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:45.025099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:45.025102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:45.025531 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:45.025831 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:45.026335 (+ 504us) server_negotiation.cc:299] Negotiation successful
0924 16:10:45.026634 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":393,"threads_started":1}
I20260924 16:10:45.028160 21254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:10:45.116863 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 21129
2026-09-24T16:10:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (66533 ms)
[----------] 30 tests from SecurityITest (324461 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:10:45 dist-test-slave-1gc3 krb5kdc[21326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:10:45 dist-test-slave-1gc3 krb5kdc[21326](info): set up 2 sockets
Sep 24 16:10:45 dist-test-slave-1gc3 krb5kdc[21326](info): commencing operation
krb5kdc: starting...
W20260924 16:10:46.187969 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.007s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:10:46 dist-test-slave-1gc3 krb5kdc[21326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:10:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:10:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:10:46.506932 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:46857
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:46857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:10:46.925246 21340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:10:46.925669 21340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:10:46.925806 21340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:10:46.934221 21340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:10:46.934384 21340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:10:46.934444 21340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:10:46.934540 21340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:10:46.934602 21340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:10:46.946614 21340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:46857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:46857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:10:46.948539 21340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:10:46.950403 21340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:10:46.965368 21345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:46.965502 21346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:46.965773 21348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:46.966836 21340 server_base.cc:1034] running on GCE node
I20260924 16:10:46.967969 21340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:10:46.970127 21340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:10:46.971433 21340 hybrid_clock.cc:648] HybridClock initialized: now 1790266246971351 us; error 65 us; skew 500 ppm
I20260924 16:10:46.971936 21340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:10:46 dist-test-slave-1gc3 krb5kdc[21326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266246, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:10:46.976022 21340 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:10:46.978291 21340 webserver.cc:460] Webserver started at http://127.3.214.254:40897/ using document root <none> and password file <none>
I20260924 16:10:46.979242 21340 fs_manager.cc:359] Metadata directory not provided
I20260924 16:10:46.979386 21340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:10:46.979801 21340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:10:46.983718 21340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "964ddd4e259a4e54828108a78d432a25"
format_stamp: "Formatted at 2026-09-24 16:10:46 on dist-test-slave-1gc3"
server_key: "6e4a9b492e5f7cb0040f2fd9dd20daa9"
server_key_iv: "1c9426d1604ab5793fa9e77957986940"
server_key_version: "encryptionkey@0"
I20260924 16:10:46.984633 21340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "964ddd4e259a4e54828108a78d432a25"
format_stamp: "Formatted at 2026-09-24 16:10:46 on dist-test-slave-1gc3"
server_key: "6e4a9b492e5f7cb0040f2fd9dd20daa9"
server_key_iv: "1c9426d1604ab5793fa9e77957986940"
server_key_version: "encryptionkey@0"
I20260924 16:10:46.991473 21340 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260924 16:10:46.996200 21355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:46.998248 21340 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260924 16:10:46.998507 21340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "964ddd4e259a4e54828108a78d432a25"
format_stamp: "Formatted at 2026-09-24 16:10:46 on dist-test-slave-1gc3"
server_key: "6e4a9b492e5f7cb0040f2fd9dd20daa9"
server_key_iv: "1c9426d1604ab5793fa9e77957986940"
server_key_version: "encryptionkey@0"
I20260924 16:10:46.998760 21340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:10:47.018349 21340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:10:47.023887 21340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:10:47.024222 21340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:10:47.046437 21340 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:46857
I20260924 16:10:47.046464 21407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:46857 every 8 connection(s)
I20260924 16:10:47.048148 21340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:10:47.052103 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 21340
I20260924 16:10:47.052316 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:10:47.052567 21408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:10:47.053138 3931 external_mini_cluster.cc:1428] Setting key 4460b1630475569a2e2505f3f70af083
I20260924 16:10:47.067425 21408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25: Bootstrap starting.
Sep 24 16:10:47 dist-test-slave-1gc3 krb5kdc[21326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:10:47.071421 21408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25: Neither blocks nor log segments found. Creating new log.
I20260924 16:10:47.072765 21408 log.cc:824] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25: Log is configured to *not* fsync() on all Append() calls
I20260924 16:10:47.076712 21408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25: No bootstrap required, opened a new log
I20260924 16:10:47.082641 21411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:47.055793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34164 (local address 127.3.214.254:46857)
0924 16:10:47.056781 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:47.056796 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:47.056852 (+ 56us) server_negotiation.cc:407] Connection header received
0924 16:10:47.059390 (+ 2538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:47.059424 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:47.060005 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:47.060512 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:47.062483 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:47.064054 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:47.065617 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:47.066573 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:47.066773 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:47.072329 (+ 5556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:47.072342 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:47.072369 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:47.072453 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:47.076673 (+ 4220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:47.077809 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:47.077818 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:47.077828 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:47.078075 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:47.078661 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:47.078667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:47.078670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:47.079552 (+ 882us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:47.079882 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:47.080519 (+ 637us) server_negotiation.cc:299] Negotiation successful
0924 16:10:47.081128 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":426,"threads_started":1}
I20260924 16:10:47.086402 21408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964ddd4e259a4e54828108a78d432a25" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 46857 } }
I20260924 16:10:47.086890 21408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:10:47.086971 21408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 964ddd4e259a4e54828108a78d432a25, State: Initialized, Role: FOLLOWER
I20260924 16:10:47.087620 21408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964ddd4e259a4e54828108a78d432a25" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 46857 } }
I20260924 16:10:47.087853 21408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:10:47.087997 21408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:10:47.088155 21408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:10:47.090775 21408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964ddd4e259a4e54828108a78d432a25" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 46857 } }
I20260924 16:10:47.091387 21408 leader_election.cc:302] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 964ddd4e259a4e54828108a78d432a25; no voters:
I20260924 16:10:47.092193 21408 leader_election.cc:288] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:10:47.092329 21413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:10:47.093286 21413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25 [term 1 LEADER]: Becoming Leader. State: Replica: 964ddd4e259a4e54828108a78d432a25, State: Running, Role: LEADER
I20260924 16:10:47.094039 21413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964ddd4e259a4e54828108a78d432a25" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 46857 } }
I20260924 16:10:47.094715 21408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:10:47.098359 21415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 964ddd4e259a4e54828108a78d432a25. Latest consensus state: current_term: 1 leader_uuid: "964ddd4e259a4e54828108a78d432a25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964ddd4e259a4e54828108a78d432a25" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 46857 } } }
I20260924 16:10:47.098868 21415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25 [sys.catalog]: This master's current role is: LEADER
I20260924 16:10:47.097673 21414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "964ddd4e259a4e54828108a78d432a25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964ddd4e259a4e54828108a78d432a25" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 46857 } } }
I20260924 16:10:47.099843 21414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25 [sys.catalog]: This master's current role is: LEADER
I20260924 16:10:47.103024 21419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:10:47.112618 21419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:10:47.126633 21419 catalog_manager.cc:1347] Generated new cluster ID: 8880d4adb1484c4f8c4f3595b7464d70
I20260924 16:10:47.126801 21419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:10:47.162802 21419 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:10:47.164548 21419 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:10:47.179211 21419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25: Generated new TSK 0
I20260924 16:10:47.180621 21419 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.214.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
I20260924 16:10:47.262368 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:46857
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:10:47.678380 21434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:10:47.678773 21434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:10:47.678910 21434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:10:47.687249 21434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:10:47.687410 21434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:10:47.687582 21434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:10:47.699951 21434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:46857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:10:47.701835 21434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:10:47.703790 21434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:10:47.720299 21442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:47.720356 21440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:47.721172 21439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:47.722002 21434 server_base.cc:1034] running on GCE node
I20260924 16:10:47.722894 21434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:10:47.724160 21434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:10:47.725474 21434 hybrid_clock.cc:648] HybridClock initialized: now 1790266247725399 us; error 57 us; skew 500 ppm
I20260924 16:10:47.725932 21434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:10:47 dist-test-slave-1gc3 krb5kdc[21326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266247, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:10:47.729768 21434 init.cc:377] Logged in from keytab as kudu/127.3.214.193@KRBTEST.COM (short username kudu)
I20260924 16:10:47.731787 21434 webserver.cc:460] Webserver started at http://127.3.214.193:43705/ using document root <none> and password file <none>
I20260924 16:10:47.732671 21434 fs_manager.cc:359] Metadata directory not provided
I20260924 16:10:47.732977 21434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:10:47.733378 21434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:10:47.737157 21434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "39a0363f23a04a1b914fb6857078f839"
format_stamp: "Formatted at 2026-09-24 16:10:47 on dist-test-slave-1gc3"
server_key: "b2648e84cee3baee3707a1950dc675c7"
server_key_iv: "fa8d37b6f163a8ab5a05b8a73dbbafbf"
server_key_version: "encryptionkey@0"
I20260924 16:10:47.738164 21434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "39a0363f23a04a1b914fb6857078f839"
format_stamp: "Formatted at 2026-09-24 16:10:47 on dist-test-slave-1gc3"
server_key: "b2648e84cee3baee3707a1950dc675c7"
server_key_iv: "fa8d37b6f163a8ab5a05b8a73dbbafbf"
server_key_version: "encryptionkey@0"
I20260924 16:10:47.745203 21434 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260924 16:10:47.750000 21449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:47.751988 21434 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:10:47.752223 21434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "39a0363f23a04a1b914fb6857078f839"
format_stamp: "Formatted at 2026-09-24 16:10:47 on dist-test-slave-1gc3"
server_key: "b2648e84cee3baee3707a1950dc675c7"
server_key_iv: "fa8d37b6f163a8ab5a05b8a73dbbafbf"
server_key_version: "encryptionkey@0"
I20260924 16:10:47.752453 21434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:10:47.766691 21434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:10:47.771296 21434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:10:47.771592 21434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:10:47.772780 21434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:10:47.774390 21434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:10:47.774474 21434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:47.774598 21434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:10:47.774652 21434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:47.812747 21434 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:34607
I20260924 16:10:47.812875 21561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:34607 every 8 connection(s)
I20260924 16:10:47.814350 21434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:10:47.820854 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 21434
I20260924 16:10:47.821128 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:10:47.822028 3931 external_mini_cluster.cc:1428] Setting key 984ea4aee4c990c41d2d8bbf27ec5fed
Sep 24 16:10:47 dist-test-slave-1gc3 krb5kdc[21326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266247, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:10:47.842976 21564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:47.817781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:49179 (local address 127.3.214.254:46857)
0924 16:10:47.818477 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:47.818485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:47.819661 (+ 1176us) server_negotiation.cc:407] Connection header received
0924 16:10:47.822990 (+ 3329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:47.822997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:47.823115 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:47.823258 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:47.826381 (+ 3123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:47.827182 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:47.829673 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:47.830224 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:47.830374 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:47.835702 (+ 5328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:47.835709 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:47.835714 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:47.835764 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:47.838751 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:47.839981 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:47.839987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:47.839990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:47.840180 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:47.840810 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:47.840816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:47.840820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:47.841199 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:47.841357 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:47.842351 (+ 994us) server_negotiation.cc:299] Negotiation successful
0924 16:10:47.842621 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":357,"threads_started":1}
I20260924 16:10:47.844339 21565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:47.818315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:46857 (local address 127.3.214.193:49179)
0924 16:10:47.819410 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:47.819481 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:47.822397 (+ 2916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:47.824007 (+ 1610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:47.824020 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:47.824935 (+ 915us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:10:47.826086 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:47.826152 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:47.827391 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:47.827398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:47.829436 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:47.829456 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:47.830551 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:47.830559 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:47.830897 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:47.831822 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:10:47.831857 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:10:47.835175 (+ 3318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:10:47.839014 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:10:47.839026 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:10:47.839051 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:10:47.839750 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:10:47.840378 (+ 628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:10:47.840383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:10:47.840386 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:10:47.840663 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:10:47.841354 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:10:47.841362 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:10:47.841860 (+ 498us) client_negotiation.cc:769] Sending connection context
0924 16:10:47.842260 (+ 400us) client_negotiation.cc:240] Negotiation successful
0924 16:10:47.842855 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":427,"threads_started":1}
I20260924 16:10:47.846727 21562 heartbeater.cc:347] Connected to a master server at 127.3.214.254:46857
I20260924 16:10:47.847245 21562 heartbeater.cc:464] Registering TS with master...
I20260924 16:10:47.848256 21562 heartbeater.cc:511] Master 127.3.214.254:46857 requested a full tablet report, sending...
I20260924 16:10:47.851411 21372 ts_manager.cc:194] Registered new tserver with Master: 39a0363f23a04a1b914fb6857078f839 (127.3.214.193:34607)
I20260924 16:10:47.853310 21372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:49179
WARNING: no policy specified for kudu/127.3.214.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
I20260924 16:10:47.897969 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:46857
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:10:48.284634 21568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:10:48.284987 21568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:10:48.285117 21568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:10:48.293813 21568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:10:48.293994 21568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:10:48.294174 21568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:10:48.305317 21568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:46857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:10:48.306866 21568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:10:48.308298 21568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:10:48.324716 21574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:48.324962 21573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:48.324759 21576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:48.325812 21568 server_base.cc:1034] running on GCE node
I20260924 16:10:48.326601 21568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:10:48.327685 21568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:10:48.328914 21568 hybrid_clock.cc:648] HybridClock initialized: now 1790266248328861 us; error 49 us; skew 500 ppm
I20260924 16:10:48.329299 21568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:10:48 dist-test-slave-1gc3 krb5kdc[21326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:10:48.332854 21568 init.cc:377] Logged in from keytab as kudu/127.3.214.194@KRBTEST.COM (short username kudu)
I20260924 16:10:48.334797 21568 webserver.cc:460] Webserver started at http://127.3.214.194:44847/ using document root <none> and password file <none>
I20260924 16:10:48.335575 21568 fs_manager.cc:359] Metadata directory not provided
I20260924 16:10:48.335716 21568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:10:48.336056 21568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:10:48.339459 21568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "624075eabc804d65889ee4dea3e4d2bf"
format_stamp: "Formatted at 2026-09-24 16:10:48 on dist-test-slave-1gc3"
server_key: "82856d60c608db858104bc26442893a3"
server_key_iv: "5ab07317ce3aa5c23ecdc394f85091e8"
server_key_version: "encryptionkey@0"
I20260924 16:10:48.340617 21568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "624075eabc804d65889ee4dea3e4d2bf"
format_stamp: "Formatted at 2026-09-24 16:10:48 on dist-test-slave-1gc3"
server_key: "82856d60c608db858104bc26442893a3"
server_key_iv: "5ab07317ce3aa5c23ecdc394f85091e8"
server_key_version: "encryptionkey@0"
I20260924 16:10:48.346740 21568 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 16:10:48.351594 21583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:48.353471 21568 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:10:48.353688 21568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "624075eabc804d65889ee4dea3e4d2bf"
format_stamp: "Formatted at 2026-09-24 16:10:48 on dist-test-slave-1gc3"
server_key: "82856d60c608db858104bc26442893a3"
server_key_iv: "5ab07317ce3aa5c23ecdc394f85091e8"
server_key_version: "encryptionkey@0"
I20260924 16:10:48.353910 21568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:10:48.392809 21568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:10:48.397434 21568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:10:48.397730 21568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:10:48.398943 21568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:10:48.400545 21568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:10:48.400640 21568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:48.400765 21568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:10:48.400820 21568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:48.439862 21568 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:42249
I20260924 16:10:48.439965 21695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:42249 every 8 connection(s)
I20260924 16:10:48.441397 21568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:10:48.443470 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 21568
I20260924 16:10:48.443666 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:10:48.444162 3931 external_mini_cluster.cc:1428] Setting key a8af474aec22f1afab2e960c6e02b989
Sep 24 16:10:48 dist-test-slave-1gc3 krb5kdc[21326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:10:48.470798 21698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:48.447006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:57239 (local address 127.3.214.254:46857)
0924 16:10:48.447634 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:48.447641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:48.448907 (+ 1266us) server_negotiation.cc:407] Connection header received
0924 16:10:48.451032 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:48.451039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:48.451147 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:48.451326 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:48.453122 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:48.454302 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:48.456216 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:48.456916 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:48.457055 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:48.462532 (+ 5477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:48.462542 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:48.462547 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:48.462612 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:48.466176 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:48.467404 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:48.467410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:48.467413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:48.467607 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:48.468228 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:48.468233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:48.468236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:48.468656 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:48.468812 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:48.469999 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0924 16:10:48.470345 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":323,"threads_started":1}
I20260924 16:10:48.472241 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:48.447749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:46857 (local address 127.3.214.194:57239)
0924 16:10:48.448699 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:48.448750 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:48.450691 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:48.451597 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:48.451607 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:48.452217 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:10:48.452889 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:48.452908 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:48.454566 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:48.454571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:48.456023 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:48.456042 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:48.457246 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:48.457254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:48.457547 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:48.458528 (+ 981us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:10:48.458563 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:10:48.462050 (+ 3487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:10:48.466466 (+ 4416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:10:48.466480 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:10:48.466505 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:10:48.467227 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:10:48.467799 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:10:48.467806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:10:48.467810 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:10:48.468082 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:10:48.468857 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:10:48.468867 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:10:48.469438 (+ 571us) client_negotiation.cc:769] Sending connection context
0924 16:10:48.469821 (+ 383us) client_negotiation.cc:240] Negotiation successful
0924 16:10:48.470577 (+ 756us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":389,"threads_started":1}
I20260924 16:10:48.474550 21696 heartbeater.cc:347] Connected to a master server at 127.3.214.254:46857
I20260924 16:10:48.475108 21696 heartbeater.cc:464] Registering TS with master...
I20260924 16:10:48.476262 21696 heartbeater.cc:511] Master 127.3.214.254:46857 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.214.195@KRBTEST.COM; defaulting to no policy
I20260924 16:10:48.478564 21372 ts_manager.cc:194] Registered new tserver with Master: 624075eabc804d65889ee4dea3e4d2bf (127.3.214.194:42249)
I20260924 16:10:48.479491 21372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:57239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
I20260924 16:10:48.518430 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:46857
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:10:48.857931 21562 heartbeater.cc:503] Master 127.3.214.254:46857 was elected leader, sending a full tablet report...
W20260924 16:10:48.884443 21702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:10:48.884786 21702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:10:48.884889 21702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:10:48.892638 21702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:10:48.892797 21702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:10:48.892953 21702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:10:48.903405 21702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:46857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:10:48.904953 21702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:10:48.906644 21702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:10:48.923274 21708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:48.923404 21707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:48.923388 21710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:48.924489 21702 server_base.cc:1034] running on GCE node
I20260924 16:10:48.925225 21702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:10:48.926515 21702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:10:48.927817 21702 hybrid_clock.cc:648] HybridClock initialized: now 1790266248927713 us; error 89 us; skew 500 ppm
I20260924 16:10:48.928257 21702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:10:48 dist-test-slave-1gc3 krb5kdc[21326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:10:48.931905 21702 init.cc:377] Logged in from keytab as kudu/127.3.214.195@KRBTEST.COM (short username kudu)
I20260924 16:10:48.934423 21702 webserver.cc:460] Webserver started at http://127.3.214.195:34271/ using document root <none> and password file <none>
I20260924 16:10:48.935247 21702 fs_manager.cc:359] Metadata directory not provided
I20260924 16:10:48.935369 21702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:10:48.935679 21702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:10:48.939195 21702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "2ae78ef2c04d43efabfc680f6e46f2e9"
format_stamp: "Formatted at 2026-09-24 16:10:48 on dist-test-slave-1gc3"
server_key: "43cb0499e680b7a13549b24228fe696e"
server_key_iv: "387fb548a9869f4723a47c071d2e2d13"
server_key_version: "encryptionkey@0"
I20260924 16:10:48.940131 21702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "2ae78ef2c04d43efabfc680f6e46f2e9"
format_stamp: "Formatted at 2026-09-24 16:10:48 on dist-test-slave-1gc3"
server_key: "43cb0499e680b7a13549b24228fe696e"
server_key_iv: "387fb548a9869f4723a47c071d2e2d13"
server_key_version: "encryptionkey@0"
I20260924 16:10:48.946720 21702 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260924 16:10:48.951941 21717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:48.954208 21702 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260924 16:10:48.954469 21702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "2ae78ef2c04d43efabfc680f6e46f2e9"
format_stamp: "Formatted at 2026-09-24 16:10:48 on dist-test-slave-1gc3"
server_key: "43cb0499e680b7a13549b24228fe696e"
server_key_iv: "387fb548a9869f4723a47c071d2e2d13"
server_key_version: "encryptionkey@0"
I20260924 16:10:48.954700 21702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:10:48.975221 21702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:10:48.980304 21702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:10:48.980597 21702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:10:48.981812 21702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:10:48.983553 21702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:10:48.983644 21702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:48.983739 21702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:10:48.983788 21702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:49.020853 21702 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:35265
I20260924 16:10:49.020928 21829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:35265 every 8 connection(s)
I20260924 16:10:49.022352 21702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:10:49.028301 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 21702
I20260924 16:10:49.028479 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:10:49.028983 3931 external_mini_cluster.cc:1428] Setting key 69e12eb3ccaa9d8b1f63986802d44344
Sep 24 16:10:49 dist-test-slave-1gc3 krb5kdc[21326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266248, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:10:49.047407 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:49.025856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:58965 (local address 127.3.214.254:46857)
0924 16:10:49.026520 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:49.026526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:49.027637 (+ 1111us) server_negotiation.cc:407] Connection header received
0924 16:10:49.030068 (+ 2431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:49.030074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:49.030208 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:49.030327 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:49.032507 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.033412 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.035257 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.035719 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.035842 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:49.040105 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:49.040111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:49.040114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:49.040159 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:49.043252 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:49.044276 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:49.044281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:49.044284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:49.044444 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:49.044953 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:49.044957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:49.044959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:49.045313 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:49.045475 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:49.046761 (+ 1286us) server_negotiation.cc:299] Negotiation successful
0924 16:10:49.047042 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":392,"threads_started":1}
I20260924 16:10:49.049212 21833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:49.026363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:46857 (local address 127.3.214.195:58965)
0924 16:10:49.027398 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:49.027462 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:49.029681 (+ 2219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:49.030682 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:49.030694 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:49.031497 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:10:49.032323 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:49.032344 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.033579 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.033583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:49.035091 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:49.035106 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.035989 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.035996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:49.036262 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:49.037028 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:10:49.037062 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:10:49.039801 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:10:49.043462 (+ 3661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:10:49.043475 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:10:49.043497 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:10:49.044107 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:10:49.044559 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:10:49.044565 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:10:49.044569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:10:49.044828 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:10:49.045446 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:10:49.045456 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:10:49.046095 (+ 639us) client_negotiation.cc:769] Sending connection context
0924 16:10:49.046544 (+ 449us) client_negotiation.cc:240] Negotiation successful
0924 16:10:49.047308 (+ 764us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":427,"threads_started":1}
I20260924 16:10:49.051507 21830 heartbeater.cc:347] Connected to a master server at 127.3.214.254:46857
I20260924 16:10:49.051987 21830 heartbeater.cc:464] Registering TS with master...
I20260924 16:10:49.053153 21830 heartbeater.cc:511] Master 127.3.214.254:46857 requested a full tablet report, sending...
I20260924 16:10:49.055985 21372 ts_manager.cc:194] Registered new tserver with Master: 2ae78ef2c04d43efabfc680f6e46f2e9 (127.3.214.195:35265)
I20260924 16:10:49.057483 21372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:58965
I20260924 16:10:49.059684 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:10:49.081310 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:49.065151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34172 (local address 127.3.214.254:46857)
0924 16:10:49.065336 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:49.065340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:49.065846 (+ 506us) server_negotiation.cc:407] Connection header received
0924 16:10:49.066116 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:49.066123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:49.066221 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:49.066356 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:49.068277 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.069154 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.070953 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.071522 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.071663 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:49.074853 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:49.074860 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:49.074865 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:49.074920 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:49.077959 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:49.079049 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:49.079054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:49.079057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:49.079261 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:49.079678 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:49.079682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:49.079684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:49.080117 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:49.080333 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:49.080806 (+ 473us) server_negotiation.cc:299] Negotiation successful
0924 16:10:49.081033 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:10:49.086804 21372 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:10:49.094699 21372 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:10:49.139358 21843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:49.111891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:42249 (local address 127.0.0.1:33084)
0924 16:10:49.127746 (+ 15855us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:49.127767 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:49.127936 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:49.129114 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:49.129126 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:49.129187 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:10:49.129811 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:49.129829 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.132326 (+ 2497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.132333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:49.135259 (+ 2926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:49.135279 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.138241 (+ 2962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.138248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:49.138480 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:49.138550 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 16:10:49.138716 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 16:10:49.138959 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15713,"thread_start_us":14257,"threads_started":1}
I20260924 16:10:49.139493 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:49.112150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33084 (local address 127.3.214.194:42249)
0924 16:10:49.127280 (+ 15130us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:49.127291 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:49.127955 (+ 664us) server_negotiation.cc:407] Connection header received
0924 16:10:49.128321 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:49.128332 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:49.128728 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:49.130141 (+ 1413us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:10:49.130454 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.132171 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.135977 (+ 3806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.137617 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.137815 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:49.138066 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:49.138841 (+ 775us) server_negotiation.cc:299] Negotiation successful
0924 16:10:49.139102 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14956,"thread_start_us":14834,"threads_started":1}
I20260924 16:10:49.140573 21847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:49.128471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39394 (local address 127.3.214.195:35265)
0924 16:10:49.130894 (+ 2423us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:49.130905 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:49.131630 (+ 725us) server_negotiation.cc:407] Connection header received
0924 16:10:49.131973 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:49.131982 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:49.132306 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:49.132509 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:10:49.133769 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.135446 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.137831 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.139060 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.139187 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:49.139469 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:49.140021 (+ 552us) server_negotiation.cc:299] Negotiation successful
0924 16:10:49.140197 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2248,"thread_start_us":2068,"threads_started":1}
I20260924 16:10:49.140558 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:49.130947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:35265 (local address 127.0.0.1:39394)
0924 16:10:49.131502 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:49.131523 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:49.131752 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:49.132853 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:49.132857 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:49.132887 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:10:49.133538 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:49.133554 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.135592 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.135611 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:49.137641 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:49.137654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.139628 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.139635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:49.139802 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:49.139837 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 16:10:49.139941 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 16:10:49.140105 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":32,"thread_start_us":296,"threads_started":1}
I20260924 16:10:49.142391 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:49.111504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:34607 (local address 127.0.0.1:35788)
0924 16:10:49.125369 (+ 13865us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:49.125460 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:49.125799 (+ 339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:49.132546 (+ 6747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:49.132550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:49.132588 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:10:49.133157 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:49.133174 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.134896 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.134901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:49.137703 (+ 2802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:49.137720 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.140144 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.141200 (+ 1056us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:49.141413 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:49.141453 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 16:10:49.141703 (+ 250us) client_negotiation.cc:240] Negotiation successful
0924 16:10:49.141889 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13335,"mutex_wait_us":1,"thread_start_us":15918,"threads_started":1}
I20260924 16:10:49.142432 21846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:49.112122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35788 (local address 127.3.214.193:34607)
0924 16:10:49.132014 (+ 19892us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:49.132022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:49.132051 (+ 29us) server_negotiation.cc:407] Connection header received
0924 16:10:49.132148 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:49.132155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:49.132326 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:49.132532 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:10:49.133447 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.134749 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.137912 (+ 3163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.139438 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.139582 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:49.139961 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:49.141776 (+ 1815us) server_negotiation.cc:299] Negotiation successful
0924 16:10:49.142021 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19662,"thread_start_us":19489,"threads_started":1}
I20260924 16:10:49.145308 21765 tablet_service.cc:1467] Processing CreateTablet for tablet 6d26d0537a904496ba970fef8c00a107 (DEFAULT_TABLE table=test-table [id=1052a6a8985c423799ef1050a10fe50d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:10:49.145239 21631 tablet_service.cc:1467] Processing CreateTablet for tablet 6d26d0537a904496ba970fef8c00a107 (DEFAULT_TABLE table=test-table [id=1052a6a8985c423799ef1050a10fe50d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:10:49.146781 21765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d26d0537a904496ba970fef8c00a107. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:10:49.147181 21631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d26d0537a904496ba970fef8c00a107. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:10:49.147519 21497 tablet_service.cc:1467] Processing CreateTablet for tablet 6d26d0537a904496ba970fef8c00a107 (DEFAULT_TABLE table=test-table [id=1052a6a8985c423799ef1050a10fe50d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:10:49.149024 21497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d26d0537a904496ba970fef8c00a107. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:10:49.160213 21849 tablet_bootstrap.cc:492] T 6d26d0537a904496ba970fef8c00a107 P 2ae78ef2c04d43efabfc680f6e46f2e9: Bootstrap starting.
I20260924 16:10:49.164265 21849 tablet_bootstrap.cc:654] T 6d26d0537a904496ba970fef8c00a107 P 2ae78ef2c04d43efabfc680f6e46f2e9: Neither blocks nor log segments found. Creating new log.
I20260924 16:10:49.165727 21850 tablet_bootstrap.cc:492] T 6d26d0537a904496ba970fef8c00a107 P 624075eabc804d65889ee4dea3e4d2bf: Bootstrap starting.
I20260924 16:10:49.166917 21849 log.cc:824] T 6d26d0537a904496ba970fef8c00a107 P 2ae78ef2c04d43efabfc680f6e46f2e9: Log is configured to *not* fsync() on all Append() calls
I20260924 16:10:49.168268 21851 tablet_bootstrap.cc:492] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839: Bootstrap starting.
I20260924 16:10:49.170531 21850 tablet_bootstrap.cc:654] T 6d26d0537a904496ba970fef8c00a107 P 624075eabc804d65889ee4dea3e4d2bf: Neither blocks nor log segments found. Creating new log.
I20260924 16:10:49.172137 21850 log.cc:824] T 6d26d0537a904496ba970fef8c00a107 P 624075eabc804d65889ee4dea3e4d2bf: Log is configured to *not* fsync() on all Append() calls
I20260924 16:10:49.172533 21851 tablet_bootstrap.cc:654] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839: Neither blocks nor log segments found. Creating new log.
I20260924 16:10:49.173691 21849 tablet_bootstrap.cc:492] T 6d26d0537a904496ba970fef8c00a107 P 2ae78ef2c04d43efabfc680f6e46f2e9: No bootstrap required, opened a new log
I20260924 16:10:49.174036 21849 ts_tablet_manager.cc:1397] T 6d26d0537a904496ba970fef8c00a107 P 2ae78ef2c04d43efabfc680f6e46f2e9: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.006s
I20260924 16:10:49.174072 21851 log.cc:824] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839: Log is configured to *not* fsync() on all Append() calls
I20260924 16:10:49.176218 21850 tablet_bootstrap.cc:492] T 6d26d0537a904496ba970fef8c00a107 P 624075eabc804d65889ee4dea3e4d2bf: No bootstrap required, opened a new log
I20260924 16:10:49.176551 21850 ts_tablet_manager.cc:1397] T 6d26d0537a904496ba970fef8c00a107 P 624075eabc804d65889ee4dea3e4d2bf: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260924 16:10:49.177281 21851 tablet_bootstrap.cc:492] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839: No bootstrap required, opened a new log
I20260924 16:10:49.177634 21851 ts_tablet_manager.cc:1397] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260924 16:10:49.186133 21851 raft_consensus.cc:348] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0363f23a04a1b914fb6857078f839" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34607 } } peers { permanent_uuid: "624075eabc804d65889ee4dea3e4d2bf" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42249 } } peers { permanent_uuid: "2ae78ef2c04d43efabfc680f6e46f2e9" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35265 } }
I20260924 16:10:49.186249 21849 raft_consensus.cc:348] T 6d26d0537a904496ba970fef8c00a107 P 2ae78ef2c04d43efabfc680f6e46f2e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0363f23a04a1b914fb6857078f839" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34607 } } peers { permanent_uuid: "624075eabc804d65889ee4dea3e4d2bf" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42249 } } peers { permanent_uuid: "2ae78ef2c04d43efabfc680f6e46f2e9" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35265 } }
I20260924 16:10:49.186726 21851 raft_consensus.cc:374] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:10:49.186825 21851 raft_consensus.cc:729] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a0363f23a04a1b914fb6857078f839, State: Initialized, Role: FOLLOWER
I20260924 16:10:49.186895 21849 raft_consensus.cc:374] T 6d26d0537a904496ba970fef8c00a107 P 2ae78ef2c04d43efabfc680f6e46f2e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:10:49.187032 21849 raft_consensus.cc:729] T 6d26d0537a904496ba970fef8c00a107 P 2ae78ef2c04d43efabfc680f6e46f2e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae78ef2c04d43efabfc680f6e46f2e9, State: Initialized, Role: FOLLOWER
I20260924 16:10:49.187426 21850 raft_consensus.cc:348] T 6d26d0537a904496ba970fef8c00a107 P 624075eabc804d65889ee4dea3e4d2bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0363f23a04a1b914fb6857078f839" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34607 } } peers { permanent_uuid: "624075eabc804d65889ee4dea3e4d2bf" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42249 } } peers { permanent_uuid: "2ae78ef2c04d43efabfc680f6e46f2e9" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35265 } }
I20260924 16:10:49.187690 21851 consensus_queue.cc:260] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0363f23a04a1b914fb6857078f839" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34607 } } peers { permanent_uuid: "624075eabc804d65889ee4dea3e4d2bf" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42249 } } peers { permanent_uuid: "2ae78ef2c04d43efabfc680f6e46f2e9" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35265 } }
I20260924 16:10:49.187944 21850 raft_consensus.cc:374] T 6d26d0537a904496ba970fef8c00a107 P 624075eabc804d65889ee4dea3e4d2bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:10:49.188108 21850 raft_consensus.cc:729] T 6d26d0537a904496ba970fef8c00a107 P 624075eabc804d65889ee4dea3e4d2bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 624075eabc804d65889ee4dea3e4d2bf, State: Initialized, Role: FOLLOWER
I20260924 16:10:49.187966 21849 consensus_queue.cc:260] T 6d26d0537a904496ba970fef8c00a107 P 2ae78ef2c04d43efabfc680f6e46f2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0363f23a04a1b914fb6857078f839" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34607 } } peers { permanent_uuid: "624075eabc804d65889ee4dea3e4d2bf" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42249 } } peers { permanent_uuid: "2ae78ef2c04d43efabfc680f6e46f2e9" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35265 } }
I20260924 16:10:49.189826 21851 ts_tablet_manager.cc:1428] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260924 16:10:49.188838 21850 consensus_queue.cc:260] T 6d26d0537a904496ba970fef8c00a107 P 624075eabc804d65889ee4dea3e4d2bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0363f23a04a1b914fb6857078f839" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34607 } } peers { permanent_uuid: "624075eabc804d65889ee4dea3e4d2bf" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42249 } } peers { permanent_uuid: "2ae78ef2c04d43efabfc680f6e46f2e9" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35265 } }
I20260924 16:10:49.192394 21696 heartbeater.cc:503] Master 127.3.214.254:46857 was elected leader, sending a full tablet report...
I20260924 16:10:49.193207 21850 ts_tablet_manager.cc:1428] T 6d26d0537a904496ba970fef8c00a107 P 624075eabc804d65889ee4dea3e4d2bf: Time spent starting tablet: real 0.017s user 0.004s sys 0.012s
W20260924 16:10:49.193974 21697 tablet.cc:2369] T 6d26d0537a904496ba970fef8c00a107 P 624075eabc804d65889ee4dea3e4d2bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:10:49.197912 21830 heartbeater.cc:503] Master 127.3.214.254:46857 was elected leader, sending a full tablet report...
I20260924 16:10:49.198974 21849 ts_tablet_manager.cc:1428] T 6d26d0537a904496ba970fef8c00a107 P 2ae78ef2c04d43efabfc680f6e46f2e9: Time spent starting tablet: real 0.025s user 0.012s sys 0.008s
I20260924 16:10:49.233247 21855 raft_consensus.cc:482] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:10:49.233589 21855 raft_consensus.cc:504] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0363f23a04a1b914fb6857078f839" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34607 } } peers { permanent_uuid: "624075eabc804d65889ee4dea3e4d2bf" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42249 } } peers { permanent_uuid: "2ae78ef2c04d43efabfc680f6e46f2e9" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35265 } }
I20260924 16:10:49.235534 21855 leader_election.cc:288] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 624075eabc804d65889ee4dea3e4d2bf (127.3.214.194:42249), 2ae78ef2c04d43efabfc680f6e46f2e9 (127.3.214.195:35265)
I20260924 16:10:49.244699 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:49.236336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:49443 (local address 127.3.214.194:42249)
0924 16:10:49.236645 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:49.236653 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:49.237351 (+ 698us) server_negotiation.cc:407] Connection header received
0924 16:10:49.237584 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:49.237590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:49.237673 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:49.237773 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:10:49.238584 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.239540 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.241882 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.243288 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.243518 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:49.243613 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:49.244211 (+ 598us) server_negotiation.cc:299] Negotiation successful
0924 16:10:49.244385 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260924 16:10:49.244699 21847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:49.236100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:35781 (local address 127.3.214.195:35265)
0924 16:10:49.236340 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:49.236346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:49.237587 (+ 1241us) server_negotiation.cc:407] Connection header received
0924 16:10:49.237793 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:49.237799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:49.237894 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:49.238008 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:10:49.238907 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.239787 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.242077 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.243370 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.243539 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:49.243613 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:49.244211 (+ 598us) server_negotiation.cc:299] Negotiation successful
0924 16:10:49.244375 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260924 16:10:49.244911 21858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:49.236108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:35265 (local address 127.3.214.193:35781)
0924 16:10:49.237430 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:49.237449 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:49.237632 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:49.238151 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:49.238157 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:49.238192 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:10:49.238698 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:49.238714 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.240040 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.240049 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:49.241858 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:49.241899 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.243696 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.243709 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:49.243990 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:49.244035 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 16:10:49.244166 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 16:10:49.244373 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1168,"mutex_wait_us":47,"thread_start_us":571,"threads_started":1}
I20260924 16:10:49.245318 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:49.236007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:42249 (local address 127.3.214.193:49443)
0924 16:10:49.237178 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:49.237206 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:49.237418 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:49.237814 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:49.237820 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:49.237855 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:10:49.238398 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:49.238411 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.239700 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.239706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:49.241668 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:49.241682 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.243732 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.243739 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:49.244022 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:49.244049 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 16:10:49.244166 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 16:10:49.244373 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":969,"mutex_wait_us":58,"thread_start_us":1043,"threads_started":1}
I20260924 16:10:49.246309 21785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d26d0537a904496ba970fef8c00a107" candidate_uuid: "39a0363f23a04a1b914fb6857078f839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae78ef2c04d43efabfc680f6e46f2e9" is_pre_election: true
I20260924 16:10:49.246937 21785 raft_consensus.cc:2454] T 6d26d0537a904496ba970fef8c00a107 P 2ae78ef2c04d43efabfc680f6e46f2e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39a0363f23a04a1b914fb6857078f839 in term 0.
I20260924 16:10:49.246876 21651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d26d0537a904496ba970fef8c00a107" candidate_uuid: "39a0363f23a04a1b914fb6857078f839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "624075eabc804d65889ee4dea3e4d2bf" is_pre_election: true
I20260924 16:10:49.247557 21651 raft_consensus.cc:2454] T 6d26d0537a904496ba970fef8c00a107 P 624075eabc804d65889ee4dea3e4d2bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39a0363f23a04a1b914fb6857078f839 in term 0.
I20260924 16:10:49.247828 21450 leader_election.cc:302] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae78ef2c04d43efabfc680f6e46f2e9, 39a0363f23a04a1b914fb6857078f839; no voters:
I20260924 16:10:49.248409 21855 raft_consensus.cc:2781] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:10:49.248579 21855 raft_consensus.cc:482] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:10:49.248687 21855 raft_consensus.cc:3037] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:10:49.251351 21855 raft_consensus.cc:504] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0363f23a04a1b914fb6857078f839" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34607 } } peers { permanent_uuid: "624075eabc804d65889ee4dea3e4d2bf" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42249 } } peers { permanent_uuid: "2ae78ef2c04d43efabfc680f6e46f2e9" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35265 } }
I20260924 16:10:49.252089 21855 leader_election.cc:288] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839 [CANDIDATE]: Term 1 election: Requested vote from peers 624075eabc804d65889ee4dea3e4d2bf (127.3.214.194:42249), 2ae78ef2c04d43efabfc680f6e46f2e9 (127.3.214.195:35265)
I20260924 16:10:49.252991 21651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d26d0537a904496ba970fef8c00a107" candidate_uuid: "39a0363f23a04a1b914fb6857078f839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "624075eabc804d65889ee4dea3e4d2bf"
I20260924 16:10:49.252991 21785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d26d0537a904496ba970fef8c00a107" candidate_uuid: "39a0363f23a04a1b914fb6857078f839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae78ef2c04d43efabfc680f6e46f2e9"
I20260924 16:10:49.253309 21785 raft_consensus.cc:3037] T 6d26d0537a904496ba970fef8c00a107 P 2ae78ef2c04d43efabfc680f6e46f2e9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:10:49.253309 21651 raft_consensus.cc:3037] T 6d26d0537a904496ba970fef8c00a107 P 624075eabc804d65889ee4dea3e4d2bf [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:10:49.257093 21651 raft_consensus.cc:2454] T 6d26d0537a904496ba970fef8c00a107 P 624075eabc804d65889ee4dea3e4d2bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39a0363f23a04a1b914fb6857078f839 in term 1.
I20260924 16:10:49.257083 21785 raft_consensus.cc:2454] T 6d26d0537a904496ba970fef8c00a107 P 2ae78ef2c04d43efabfc680f6e46f2e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39a0363f23a04a1b914fb6857078f839 in term 1.
I20260924 16:10:49.257753 21450 leader_election.cc:302] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2ae78ef2c04d43efabfc680f6e46f2e9, 39a0363f23a04a1b914fb6857078f839, 624075eabc804d65889ee4dea3e4d2bf; no voters:
I20260924 16:10:49.258216 21855 raft_consensus.cc:2781] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:10:49.258889 21855 raft_consensus.cc:686] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839 [term 1 LEADER]: Becoming Leader. State: Replica: 39a0363f23a04a1b914fb6857078f839, State: Running, Role: LEADER
I20260924 16:10:49.259639 21855 consensus_queue.cc:237] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0363f23a04a1b914fb6857078f839" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34607 } } peers { permanent_uuid: "624075eabc804d65889ee4dea3e4d2bf" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42249 } } peers { permanent_uuid: "2ae78ef2c04d43efabfc680f6e46f2e9" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35265 } }
I20260924 16:10:49.265708 21371 catalog_manager.cc:5495] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39a0363f23a04a1b914fb6857078f839 (127.3.214.193). New cstate: current_term: 1 leader_uuid: "39a0363f23a04a1b914fb6857078f839" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0363f23a04a1b914fb6857078f839" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 34607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "624075eabc804d65889ee4dea3e4d2bf" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae78ef2c04d43efabfc680f6e46f2e9" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35265 } health_report { overall_health: UNKNOWN } } }
W20260924 16:10:49.274384 21831 tablet.cc:2369] T 6d26d0537a904496ba970fef8c00a107 P 2ae78ef2c04d43efabfc680f6e46f2e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:10:49.288005 21846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:49.281642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35804 (local address 127.3.214.193:34607)
0924 16:10:49.281874 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:49.281879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:49.281900 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:10:49.282128 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:49.282133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:49.282251 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:49.282469 (+ 218us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:10:49.282904 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.283712 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.285351 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.285804 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.285922 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:49.286623 (+ 701us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:10:49.287351 (+ 728us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:10:49.287484 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:49.287635 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 16:10:49.287725 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:10:49.298904 21785 raft_consensus.cc:1261] T 6d26d0537a904496ba970fef8c00a107 P 2ae78ef2c04d43efabfc680f6e46f2e9 [term 1 FOLLOWER]: Refusing update from remote peer 39a0363f23a04a1b914fb6857078f839: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:10:49.298904 21651 raft_consensus.cc:1261] T 6d26d0537a904496ba970fef8c00a107 P 624075eabc804d65889ee4dea3e4d2bf [term 1 FOLLOWER]: Refusing update from remote peer 39a0363f23a04a1b914fb6857078f839: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:10:49.300323 21860 consensus_queue.cc:1035] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "624075eabc804d65889ee4dea3e4d2bf" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 42249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:10:49.300812 21855 consensus_queue.cc:1035] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae78ef2c04d43efabfc680f6e46f2e9" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:10:49.315392 21863 mvcc.cc:205] Tried to move back new op lower bound from 7332930557114093568 to 7332930556972523520. Current Snapshot: MvccSnapshot[applied={T|T < 7332930557114093568}]
I20260924 16:10:49.319639 21864 mvcc.cc:205] Tried to move back new op lower bound from 7332930557114093568 to 7332930556972523520. Current Snapshot: MvccSnapshot[applied={T|T < 7332930557114093568}]
I20260924 16:10:49.332418 21372 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:10:49.332778 21372 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:10:49.337164 21372 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 964ddd4e259a4e54828108a78d432a25: Sending DeleteTablet for 3 replicas of tablet 6d26d0537a904496ba970fef8c00a107
I20260924 16:10:49.338951 21765 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d26d0537a904496ba970fef8c00a107 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:10:49 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:39394
I20260924 16:10:49.339016 21631 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d26d0537a904496ba970fef8c00a107 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:10:49 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:33084
I20260924 16:10:49.339298 21497 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d26d0537a904496ba970fef8c00a107 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:10:49 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:35788
I20260924 16:10:49.343096 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 21434
I20260924 16:10:49.349802 21874 tablet_replica.cc:330] T 6d26d0537a904496ba970fef8c00a107 P 2ae78ef2c04d43efabfc680f6e46f2e9: stopping tablet replica
I20260924 16:10:49.350852 21875 tablet_replica.cc:330] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839: stopping tablet replica
I20260924 16:10:49.351248 21874 raft_consensus.cc:2229] T 6d26d0537a904496ba970fef8c00a107 P 2ae78ef2c04d43efabfc680f6e46f2e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:10:49.351882 21874 raft_consensus.cc:2258] T 6d26d0537a904496ba970fef8c00a107 P 2ae78ef2c04d43efabfc680f6e46f2e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:10:49.352512 21875 raft_consensus.cc:2229] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:10:49.354527 21874 ts_tablet_manager.cc:1905] T 6d26d0537a904496ba970fef8c00a107 P 2ae78ef2c04d43efabfc680f6e46f2e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:10:49.355937 21873 tablet_replica.cc:330] T 6d26d0537a904496ba970fef8c00a107 P 624075eabc804d65889ee4dea3e4d2bf: stopping tablet replica
I20260924 16:10:49.356823 21873 raft_consensus.cc:2229] T 6d26d0537a904496ba970fef8c00a107 P 624075eabc804d65889ee4dea3e4d2bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:10:49.357643 21875 raft_consensus.cc:2258] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:10:49.357715 21873 raft_consensus.cc:2258] T 6d26d0537a904496ba970fef8c00a107 P 624075eabc804d65889ee4dea3e4d2bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:10:49.360689 21875 ts_tablet_manager.cc:1905] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:10:49.361112 21873 ts_tablet_manager.cc:1905] T 6d26d0537a904496ba970fef8c00a107 P 624075eabc804d65889ee4dea3e4d2bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:10:49.363078 21874 ts_tablet_manager.cc:1918] T 6d26d0537a904496ba970fef8c00a107 P 2ae78ef2c04d43efabfc680f6e46f2e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:10:49.363384 21874 log.cc:1196] T 6d26d0537a904496ba970fef8c00a107 P 2ae78ef2c04d43efabfc680f6e46f2e9: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/6d26d0537a904496ba970fef8c00a107
I20260924 16:10:49.364104 21874 ts_tablet_manager.cc:1939] T 6d26d0537a904496ba970fef8c00a107 P 2ae78ef2c04d43efabfc680f6e46f2e9: Deleting consensus metadata
Sep 24 16:10:49 dist-test-slave-1gc3 krb5kdc[21326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.193@KRBTEST.COM
I20260924 16:10:49.366657 21356 catalog_manager.cc:4841] TS 2ae78ef2c04d43efabfc680f6e46f2e9 (127.3.214.195:35265): tablet 6d26d0537a904496ba970fef8c00a107 (table test-table [id=1052a6a8985c423799ef1050a10fe50d]) successfully deleted
I20260924 16:10:49.368782 21875 ts_tablet_manager.cc:1918] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:10:49.369112 21875 log.cc:1196] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-0/wal/wals/6d26d0537a904496ba970fef8c00a107
I20260924 16:10:49.370211 21873 ts_tablet_manager.cc:1918] T 6d26d0537a904496ba970fef8c00a107 P 624075eabc804d65889ee4dea3e4d2bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:10:49.370548 21873 log.cc:1196] T 6d26d0537a904496ba970fef8c00a107 P 624075eabc804d65889ee4dea3e4d2bf: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/6d26d0537a904496ba970fef8c00a107
I20260924 16:10:49.369823 21875 ts_tablet_manager.cc:1939] T 6d26d0537a904496ba970fef8c00a107 P 39a0363f23a04a1b914fb6857078f839: Deleting consensus metadata
I20260924 16:10:49.371062 21873 ts_tablet_manager.cc:1939] T 6d26d0537a904496ba970fef8c00a107 P 624075eabc804d65889ee4dea3e4d2bf: Deleting consensus metadata
I20260924 16:10:49.373541 21356 catalog_manager.cc:4841] TS 39a0363f23a04a1b914fb6857078f839 (127.3.214.193:34607): tablet 6d26d0537a904496ba970fef8c00a107 (table test-table [id=1052a6a8985c423799ef1050a10fe50d]) successfully deleted
I20260924 16:10:49.373663 21359 catalog_manager.cc:4841] TS 624075eabc804d65889ee4dea3e4d2bf (127.3.214.194:42249): tablet 6d26d0537a904496ba970fef8c00a107 (table test-table [id=1052a6a8985c423799ef1050a10fe50d]) successfully deleted
I20260924 16:10:49.375484 21846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:49.344609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35818 (local address 127.3.214.193:34607)
0924 16:10:49.344905 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:49.344912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:49.351237 (+ 6325us) server_negotiation.cc:407] Connection header received
0924 16:10:49.351466 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:49.351473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:49.351597 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:49.351728 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:49.353971 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.355423 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.357540 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.358357 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.358471 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:49.366061 (+ 7590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:49.366091 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:49.366179 (+ 88us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:49.366251 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:49.371300 (+ 5049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:49.372574 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:49.372584 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:49.372587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:49.372825 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:49.373367 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:49.373372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:49.373375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:49.373946 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:49.374407 (+ 461us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:49.374809 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 16:10:49.375102 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260924 16:10:49.376464 21557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:10:49.486577 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 21434
I20260924 16:10:49.505905 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 21568
Sep 24 16:10:49 dist-test-slave-1gc3 krb5kdc[21326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.194@KRBTEST.COM
I20260924 16:10:49.523867 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:49.506916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33096 (local address 127.3.214.194:42249)
0924 16:10:49.507116 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:49.507121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:49.507204 (+ 83us) server_negotiation.cc:407] Connection header received
0924 16:10:49.507375 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:49.507380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:49.507477 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:49.507586 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:49.508940 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.509796 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.511342 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.511823 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.511942 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:49.516931 (+ 4989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:49.516980 (+ 49us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:49.517006 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:49.517062 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:49.520352 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:49.521301 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:49.521309 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:49.521312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:49.521454 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:49.521857 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:49.521861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:49.521863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:49.522495 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:49.522843 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:49.523201 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 16:10:49.523465 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:10:49.524605 21691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:10:49.619774 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 21568
I20260924 16:10:49.641709 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 21702
Sep 24 16:10:49 dist-test-slave-1gc3 krb5kdc[21326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.195@KRBTEST.COM
I20260924 16:10:49.661746 21847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:49.642672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39408 (local address 127.3.214.195:35265)
0924 16:10:49.642942 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:49.642947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:49.642972 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:10:49.643140 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:49.643145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:49.643244 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:49.643332 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:49.644786 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.645590 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.647068 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:49.647483 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:49.647579 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:49.652912 (+ 5333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:49.652937 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:49.652962 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:49.653012 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:49.657235 (+ 4223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:49.658511 (+ 1276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:49.658528 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:49.658539 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:49.658730 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:49.659250 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:49.659256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:49.659259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:49.659855 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:49.660242 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:49.661111 (+ 869us) server_negotiation.cc:299] Negotiation successful
0924 16:10:49.661393 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 16:10:49.662606 21825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:10:49.762852 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 21702
I20260924 16:10:49.783032 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 21340
I20260924 16:10:49.784471 21403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:10:49.903106 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 21340
2026-09-24T16:10:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4787 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T16:10:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:10:49Z Disabled control of system clock
I20260924 16:10:49.992825 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:40895
--webserver_interface=127.3.214.254
--webserver_port=0
--builtin_ntp_servers=127.3.214.212:34511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:40895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:10:50.353773 21884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:10:50.354149 21884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:10:50.354280 21884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:10:50.362043 21884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:10:50.362236 21884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:10:50.362337 21884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:10:50.362468 21884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:10:50.362546 21884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:10:50.373006 21884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:34511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:40895
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:40895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:10:50.374512 21884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:10:50.376508 21884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:10:50.390022 21892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:50.390141 21889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:50.390137 21890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:50.391521 21884 server_base.cc:1034] running on GCE node
I20260924 16:10:50.392633 21884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:10:50.394737 21884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:10:50.396020 21884 hybrid_clock.cc:648] HybridClock initialized: now 1790266250395932 us; error 70 us; skew 500 ppm
I20260924 16:10:50.396538 21884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:10:50.406811 21884 webserver.cc:460] Webserver started at http://127.3.214.254:40345/ using document root <none> and password file <none>
I20260924 16:10:50.407817 21884 fs_manager.cc:359] Metadata directory not provided
I20260924 16:10:50.407955 21884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:10:50.408358 21884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:10:50.412211 21884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "74313c9778cc4e35a99eba644f45bb58"
format_stamp: "Formatted at 2026-09-24 16:10:50 on dist-test-slave-1gc3"
server_key: "a165fae11cc852bef6ba106a4509536c"
server_key_iv: "515888c2a73b450dd4c4af3cbcae8367"
server_key_version: "encryptionkey@0"
I20260924 16:10:50.413236 21884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "74313c9778cc4e35a99eba644f45bb58"
format_stamp: "Formatted at 2026-09-24 16:10:50 on dist-test-slave-1gc3"
server_key: "a165fae11cc852bef6ba106a4509536c"
server_key_iv: "515888c2a73b450dd4c4af3cbcae8367"
server_key_version: "encryptionkey@0"
I20260924 16:10:50.420253 21884 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260924 16:10:50.425254 21898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:50.427347 21884 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:10:50.427606 21884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "74313c9778cc4e35a99eba644f45bb58"
format_stamp: "Formatted at 2026-09-24 16:10:50 on dist-test-slave-1gc3"
server_key: "a165fae11cc852bef6ba106a4509536c"
server_key_iv: "515888c2a73b450dd4c4af3cbcae8367"
server_key_version: "encryptionkey@0"
I20260924 16:10:50.427887 21884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:10:50.444979 21884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:10:50.446230 21884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:10:50.446568 21884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:10:50.468825 21884 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:40895
I20260924 16:10:50.468854 21949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:40895 every 8 connection(s)
I20260924 16:10:50.470642 21884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:10:50.474603 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 21884
I20260924 16:10:50.474815 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:10:50.475410 3931 external_mini_cluster.cc:1428] Setting key 8b4fd0cb36e27894dc903a406f237946
I20260924 16:10:50.475664 21950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:10:50.492453 21950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58: Bootstrap starting.
I20260924 16:10:50.492966 21953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:50.478017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35630 (local address 127.3.214.254:40895)
0924 16:10:50.479377 (+ 1360us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:50.479395 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:50.479455 (+ 60us) server_negotiation.cc:407] Connection header received
0924 16:10:50.481514 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:50.481544 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:50.482243 (+ 699us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:50.482698 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:50.483497 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:50.485089 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:50.487195 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:50.488027 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:50.488186 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:50.489705 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:50.489714 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:50.489736 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:10:50.490545 (+ 809us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:50.490659 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:10:50.490685 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:50.490798 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:50.491111 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 16:10:50.491611 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":427,"threads_started":1}
I20260924 16:10:50.496464 21950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58: Neither blocks nor log segments found. Creating new log.
I20260924 16:10:50.497653 21950 log.cc:824] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58: Log is configured to *not* fsync() on all Append() calls
I20260924 16:10:50.501242 21950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58: No bootstrap required, opened a new log
I20260924 16:10:50.509544 21950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74313c9778cc4e35a99eba644f45bb58" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40895 } }
I20260924 16:10:50.510221 21950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:10:50.510349 21950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74313c9778cc4e35a99eba644f45bb58, State: Initialized, Role: FOLLOWER
I20260924 16:10:50.511068 21950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74313c9778cc4e35a99eba644f45bb58" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40895 } }
I20260924 16:10:50.511348 21950 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:10:50.511488 21950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:10:50.511665 21950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:10:50.514313 21950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74313c9778cc4e35a99eba644f45bb58" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40895 } }
I20260924 16:10:50.515153 21950 leader_election.cc:302] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74313c9778cc4e35a99eba644f45bb58; no voters:
I20260924 16:10:50.515911 21950 leader_election.cc:288] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:10:50.516203 21955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:10:50.517063 21955 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58 [term 1 LEADER]: Becoming Leader. State: Replica: 74313c9778cc4e35a99eba644f45bb58, State: Running, Role: LEADER
I20260924 16:10:50.517766 21955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74313c9778cc4e35a99eba644f45bb58" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40895 } }
I20260924 16:10:50.518585 21950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:10:50.521516 21956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74313c9778cc4e35a99eba644f45bb58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74313c9778cc4e35a99eba644f45bb58" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40895 } } }
I20260924 16:10:50.521430 21957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74313c9778cc4e35a99eba644f45bb58. Latest consensus state: current_term: 1 leader_uuid: "74313c9778cc4e35a99eba644f45bb58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74313c9778cc4e35a99eba644f45bb58" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40895 } } }
I20260924 16:10:50.522572 21957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58 [sys.catalog]: This master's current role is: LEADER
I20260924 16:10:50.522977 21956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58 [sys.catalog]: This master's current role is: LEADER
W20260924 16:10:50.533311 21971 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:10:50.533564 21971 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:10:50.534282 21963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:10:50.536178 21963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:10:50.547161 21963 catalog_manager.cc:1347] Generated new cluster ID: a191460a84794625841d4ce8bf9ca339
I20260924 16:10:50.547350 21963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:10:50.572588 21963 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:10:50.574076 21963 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:10:50.585027 21963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58: Generated new TSK 0
I20260924 16:10:50.586027 21963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:10:50.602632 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:40895
--builtin_ntp_servers=127.3.214.212:34511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:10:50.963662 21974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:10:50.964223 21974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:10:50.964383 21974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:10:50.972214 21974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:10:50.972359 21974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:10:50.972502 21974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:10:50.982832 21974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:34511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:40895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:10:50.984556 21974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:10:50.986665 21974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:10:51.002678 21979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:51.002700 21980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:51.002924 21982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:51.003561 21974 server_base.cc:1034] running on GCE node
I20260924 16:10:51.004518 21974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:10:51.005898 21974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:10:51.007190 21974 hybrid_clock.cc:648] HybridClock initialized: now 1790266251007118 us; error 50 us; skew 500 ppm
I20260924 16:10:51.007674 21974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:10:51.010918 21974 webserver.cc:460] Webserver started at http://127.3.214.193:39209/ using document root <none> and password file <none>
I20260924 16:10:51.011861 21974 fs_manager.cc:359] Metadata directory not provided
I20260924 16:10:51.011987 21974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:10:51.012414 21974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:10:51.016299 21974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "21ce193f023c45ba997b76609635e23d"
format_stamp: "Formatted at 2026-09-24 16:10:51 on dist-test-slave-1gc3"
server_key: "e370a39162374d320adcfcddfe55d467"
server_key_iv: "c24ddaa7ac14cbaf31a0cedb7e37ceb2"
server_key_version: "encryptionkey@0"
I20260924 16:10:51.017307 21974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "21ce193f023c45ba997b76609635e23d"
format_stamp: "Formatted at 2026-09-24 16:10:51 on dist-test-slave-1gc3"
server_key: "e370a39162374d320adcfcddfe55d467"
server_key_iv: "c24ddaa7ac14cbaf31a0cedb7e37ceb2"
server_key_version: "encryptionkey@0"
I20260924 16:10:51.024516 21974 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260924 16:10:51.029407 21988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:51.031358 21974 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 16:10:51.031581 21974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "21ce193f023c45ba997b76609635e23d"
format_stamp: "Formatted at 2026-09-24 16:10:51 on dist-test-slave-1gc3"
server_key: "e370a39162374d320adcfcddfe55d467"
server_key_iv: "c24ddaa7ac14cbaf31a0cedb7e37ceb2"
server_key_version: "encryptionkey@0"
I20260924 16:10:51.031899 21974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:10:51.052826 21974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:10:51.053944 21974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:10:51.054327 21974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:10:51.055567 21974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:10:51.057435 21974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:10:51.057565 21974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:51.057682 21974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:10:51.057760 21974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:51.097033 21974 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:43425
I20260924 16:10:51.097083 22101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:43425 every 8 connection(s)
I20260924 16:10:51.098917 21974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:10:51.101785 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 21974
I20260924 16:10:51.101979 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:10:51.102612 3931 external_mini_cluster.cc:1428] Setting key c95a89bb481d671820f6d6f7d47ffe4d
I20260924 16:10:51.106768 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:40895
--builtin_ntp_servers=127.3.214.212:34511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:10:51.117584 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:51.102609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:47069 (local address 127.3.214.254:40895)
0924 16:10:51.103292 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:51.103299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:51.105121 (+ 1822us) server_negotiation.cc:407] Connection header received
0924 16:10:51.108249 (+ 3128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:51.108255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:51.108352 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:51.108505 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:51.110572 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:51.111352 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:51.113872 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:51.114404 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:51.114548 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:51.116599 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:51.116604 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:51.116608 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:10:51.116635 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:51.116702 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:10:51.116718 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:51.116823 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:51.117143 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 16:10:51.117261 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":322,"threads_started":1}
I20260924 16:10:51.119423 22105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:51.103310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40895 (local address 127.3.214.193:47069)
0924 16:10:51.104917 (+ 1607us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:51.104992 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:51.107698 (+ 2706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:51.108935 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:51.108953 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:51.109137 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:10:51.110258 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:51.110292 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:51.111562 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:51.111567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:51.113621 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:51.113640 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:51.114703 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:51.114709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:51.115052 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:51.116052 (+ 1000us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:10:51.116056 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:10:51.116130 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:10:51.116287 (+ 157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:10:51.116838 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:10:51.116849 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:10:51.116857 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 16:10:51.117083 (+ 226us) client_negotiation.cc:240] Negotiation successful
0924 16:10:51.117651 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":404,"threads_started":1}
I20260924 16:10:51.122015 22102 heartbeater.cc:347] Connected to a master server at 127.3.214.254:40895
I20260924 16:10:51.122579 22102 heartbeater.cc:464] Registering TS with master...
I20260924 16:10:51.123656 22102 heartbeater.cc:511] Master 127.3.214.254:40895 requested a full tablet report, sending...
I20260924 16:10:51.128643 21915 ts_manager.cc:194] Registered new tserver with Master: 21ce193f023c45ba997b76609635e23d (127.3.214.193:43425)
I20260924 16:10:51.130775 21915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.214.193:47069
W20260924 16:10:51.484582 22106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:10:51.484915 22106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:10:51.485018 22106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:10:51.494305 22106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:10:51.494438 22106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:10:51.494563 22106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:10:51.505512 22106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:34511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:40895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:10:51.507019 22106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:10:51.509179 22106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:10:51.525102 22111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:51.525142 22112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:51.525206 22114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:51.526314 22106 server_base.cc:1034] running on GCE node
I20260924 16:10:51.527050 22106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:10:51.528225 22106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:10:51.529507 22106 hybrid_clock.cc:648] HybridClock initialized: now 1790266251529448 us; error 47 us; skew 500 ppm
I20260924 16:10:51.529947 22106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:10:51.533581 22106 webserver.cc:460] Webserver started at http://127.3.214.194:46431/ using document root <none> and password file <none>
I20260924 16:10:51.534680 22106 fs_manager.cc:359] Metadata directory not provided
I20260924 16:10:51.534812 22106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:10:51.535200 22106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:10:51.539058 22106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "9ed58339f0e442638f54286f7ff4226b"
format_stamp: "Formatted at 2026-09-24 16:10:51 on dist-test-slave-1gc3"
server_key: "9d2dbbf7c1e50b4c1dae1f8f324c7f14"
server_key_iv: "3bc2f67f8de42449f6e04b90fe93d776"
server_key_version: "encryptionkey@0"
I20260924 16:10:51.540109 22106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "9ed58339f0e442638f54286f7ff4226b"
format_stamp: "Formatted at 2026-09-24 16:10:51 on dist-test-slave-1gc3"
server_key: "9d2dbbf7c1e50b4c1dae1f8f324c7f14"
server_key_iv: "3bc2f67f8de42449f6e04b90fe93d776"
server_key_version: "encryptionkey@0"
I20260924 16:10:51.547137 22106 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260924 16:10:51.552333 22120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:51.554220 22106 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260924 16:10:51.554453 22106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "9ed58339f0e442638f54286f7ff4226b"
format_stamp: "Formatted at 2026-09-24 16:10:51 on dist-test-slave-1gc3"
server_key: "9d2dbbf7c1e50b4c1dae1f8f324c7f14"
server_key_iv: "3bc2f67f8de42449f6e04b90fe93d776"
server_key_version: "encryptionkey@0"
I20260924 16:10:51.554728 22106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:10:51.595017 22106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:10:51.596180 22106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:10:51.596531 22106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:10:51.597797 22106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:10:51.599804 22106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:10:51.599922 22106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:51.600042 22106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:10:51.600116 22106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:51.638486 22106 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:45947
I20260924 16:10:51.638549 22233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:45947 every 8 connection(s)
I20260924 16:10:51.640218 22106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:10:51.649302 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 22106
I20260924 16:10:51.649511 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:10:51.650161 3931 external_mini_cluster.cc:1428] Setting key b70791ddebcf2166378435a51866553e
I20260924 16:10:51.653874 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:40895
--builtin_ntp_servers=127.3.214.212:34511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:10:51.656898 22236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:51.643567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:57041 (local address 127.3.214.254:40895)
0924 16:10:51.644205 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:51.644211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:51.645948 (+ 1737us) server_negotiation.cc:407] Connection header received
0924 16:10:51.648160 (+ 2212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:51.648166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:51.648261 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:51.648362 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:51.650048 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:51.651113 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:51.653256 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:51.653759 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:51.653872 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:51.655966 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:51.655971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:51.655975 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:10:51.656001 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:51.656060 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:10:51.656074 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:51.656157 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:51.656446 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 16:10:51.656569 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":322,"threads_started":1}
I20260924 16:10:51.658532 22237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:51.644400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40895 (local address 127.3.214.194:57041)
0924 16:10:51.645706 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:51.645759 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:51.647799 (+ 2040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:51.648664 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:51.648673 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:51.648787 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:10:51.649750 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:51.649781 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:51.651343 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:51.651349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:51.653045 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:51.653065 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:51.654094 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:51.654142 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:51.654476 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:51.655357 (+ 881us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:10:51.655362 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:10:51.655428 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:10:51.655676 (+ 248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:10:51.656194 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:10:51.656205 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:10:51.656214 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 16:10:51.656435 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 16:10:51.656960 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":366,"threads_started":1}
I20260924 16:10:51.660842 22234 heartbeater.cc:347] Connected to a master server at 127.3.214.254:40895
I20260924 16:10:51.661309 22234 heartbeater.cc:464] Registering TS with master...
I20260924 16:10:51.662437 22234 heartbeater.cc:511] Master 127.3.214.254:40895 requested a full tablet report, sending...
I20260924 16:10:51.664783 21915 ts_manager.cc:194] Registered new tserver with Master: 9ed58339f0e442638f54286f7ff4226b (127.3.214.194:45947)
I20260924 16:10:51.665733 21915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.214.194:57041
W20260924 16:10:52.069272 22238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:10:52.069705 22238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:10:52.069851 22238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:10:52.078315 22238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:10:52.078495 22238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:10:52.078668 22238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:10:52.089941 22238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:34511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:40895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:10:52.091817 22238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:10:52.094246 22238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:10:52.111289 22243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:52.111414 22244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:52.111323 22246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:52.116576 22238 server_base.cc:1034] running on GCE node
I20260924 16:10:52.117558 22238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:10:52.119302 22238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:10:52.120939 22238 hybrid_clock.cc:648] HybridClock initialized: now 1790266252120756 us; error 155 us; skew 500 ppm
I20260924 16:10:52.121543 22238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:10:52.128437 22238 webserver.cc:460] Webserver started at http://127.3.214.195:46011/ using document root <none> and password file <none>
I20260924 16:10:52.129650 22238 fs_manager.cc:359] Metadata directory not provided
I20260924 16:10:52.129830 22238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:10:52.130424 22238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:10:52.135347 22238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "57697a03d0a54f2c9943b81cafc275f0"
format_stamp: "Formatted at 2026-09-24 16:10:52 on dist-test-slave-1gc3"
server_key: "0f60171226d85f49e82b9fb6f062a58b"
server_key_iv: "4b792294051557be8e665289837212c7"
server_key_version: "encryptionkey@0"
I20260924 16:10:52.135769 22102 heartbeater.cc:503] Master 127.3.214.254:40895 was elected leader, sending a full tablet report...
I20260924 16:10:52.136596 22238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "57697a03d0a54f2c9943b81cafc275f0"
format_stamp: "Formatted at 2026-09-24 16:10:52 on dist-test-slave-1gc3"
server_key: "0f60171226d85f49e82b9fb6f062a58b"
server_key_iv: "4b792294051557be8e665289837212c7"
server_key_version: "encryptionkey@0"
I20260924 16:10:52.145334 22238 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260924 16:10:52.151189 22252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:52.153440 22238 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260924 16:10:52.153726 22238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "57697a03d0a54f2c9943b81cafc275f0"
format_stamp: "Formatted at 2026-09-24 16:10:52 on dist-test-slave-1gc3"
server_key: "0f60171226d85f49e82b9fb6f062a58b"
server_key_iv: "4b792294051557be8e665289837212c7"
server_key_version: "encryptionkey@0"
I20260924 16:10:52.154060 22238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:10:52.171635 22238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:10:52.172860 22238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:10:52.173259 22238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:10:52.174667 22238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:10:52.176856 22238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:10:52.176980 22238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:52.177109 22238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:10:52.177196 22238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:52.221248 22238 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:32821
I20260924 16:10:52.221355 22366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:32821 every 8 connection(s)
I20260924 16:10:52.223248 22238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:10:52.232883 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 22238
I20260924 16:10:52.233093 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:10:52.233686 3931 external_mini_cluster.cc:1428] Setting key 254a3d380cf27563c201b59cda488fa1
I20260924 16:10:52.242934 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:52.227021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:58683 (local address 127.3.214.254:40895)
0924 16:10:52.227695 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:52.227701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:52.229904 (+ 2203us) server_negotiation.cc:407] Connection header received
0924 16:10:52.232796 (+ 2892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:52.232805 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:52.232945 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:52.233124 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:52.235475 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.236735 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.239098 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.239813 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.239970 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:52.241658 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:52.241665 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:52.241672 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:10:52.241705 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:52.241777 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:10:52.241799 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:52.241909 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:52.242290 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 16:10:52.242441 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":342,"threads_started":1}
I20260924 16:10:52.244009 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:52.228014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:40895 (local address 127.3.214.195:58683)
0924 16:10:52.229687 (+ 1673us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:52.229764 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:52.232338 (+ 2574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:52.233655 (+ 1317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:52.233672 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:52.233854 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:10:52.235044 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:52.235086 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.236985 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.236993 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:52.238907 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:52.238932 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.240086 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.240093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:52.240372 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:52.241230 (+ 858us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:10:52.241233 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:10:52.241283 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:10:52.241392 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:10:52.241969 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:10:52.241983 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:10:52.241993 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 16:10:52.242233 (+ 240us) client_negotiation.cc:240] Negotiation successful
0924 16:10:52.242715 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":422,"threads_started":1}
I20260924 16:10:52.245929 22367 heartbeater.cc:347] Connected to a master server at 127.3.214.254:40895
I20260924 16:10:52.246330 22367 heartbeater.cc:464] Registering TS with master...
I20260924 16:10:52.247212 22367 heartbeater.cc:511] Master 127.3.214.254:40895 requested a full tablet report, sending...
I20260924 16:10:52.249402 21914 ts_manager.cc:194] Registered new tserver with Master: 57697a03d0a54f2c9943b81cafc275f0 (127.3.214.195:32821)
I20260924 16:10:52.250648 21914 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.214.195:58683
I20260924 16:10:52.250995 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:10:52.267637 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:52.260698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.3.214.254:40895)
0924 16:10:52.261011 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:52.261019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:52.261306 (+ 287us) server_negotiation.cc:407] Connection header received
0924 16:10:52.261548 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:52.261556 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:52.261644 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:52.261751 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:52.262423 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.263287 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.264794 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.265244 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.265370 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:52.266666 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:52.266671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:52.266675 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:10:52.266713 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:52.266755 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:10:52.266775 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:52.266979 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:52.267101 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 16:10:52.267269 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 16:10:52.273043 21914 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:10:52.280385 21914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:10:52.308686 22380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:52.295728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:43425 (local address 127.0.0.1:52758)
0924 16:10:52.297853 (+ 2125us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:52.297895 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:52.298164 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:52.299460 (+ 1296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:52.299466 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:52.299503 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:10:52.300188 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:52.300203 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.302597 (+ 2394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.302610 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:52.305291 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:52.305305 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.307558 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.307564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:52.307788 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:52.307841 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 16:10:52.307985 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 16:10:52.308241 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1793,"spinlock_wait_cycles":2304,"thread_start_us":1556,"threads_started":1}
I20260924 16:10:52.309386 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:52.296510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40522 (local address 127.3.214.194:45947)
0924 16:10:52.297382 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:52.297406 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:52.298142 (+ 736us) server_negotiation.cc:407] Connection header received
0924 16:10:52.298464 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:52.298470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:52.298761 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:52.298972 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:10:52.300463 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.302373 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.306134 (+ 3761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.307598 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.307737 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:52.308030 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:52.308737 (+ 707us) server_negotiation.cc:299] Negotiation successful
0924 16:10:52.309003 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":613,"threads_started":1}
I20260924 16:10:52.309652 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:52.296096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52758 (local address 127.3.214.193:43425)
0924 16:10:52.296940 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:52.296951 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:52.298037 (+ 1086us) server_negotiation.cc:407] Connection header received
0924 16:10:52.298516 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:52.298527 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:52.298844 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:52.299072 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:10:52.300430 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.302378 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.305494 (+ 3116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.306884 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.307062 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:52.307424 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:52.308984 (+ 1560us) server_negotiation.cc:299] Negotiation successful
0924 16:10:52.309169 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":406,"threads_started":1}
I20260924 16:10:52.309767 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:52.296160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:45947 (local address 127.0.0.1:40522)
0924 16:10:52.297851 (+ 1691us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:52.297888 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:52.298168 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:52.299292 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:52.299303 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:52.299341 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:10:52.299999 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:52.300132 (+ 133us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.302608 (+ 2476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.302617 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:52.305853 (+ 3236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:52.305870 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.308193 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.308199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:52.308428 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:52.308485 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 16:10:52.308612 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 16:10:52.309189 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1498,"mutex_wait_us":119,"spinlock_wait_cycles":29056,"thread_start_us":2006,"threads_started":1}
I20260924 16:10:52.313176 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:52.298669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:32821 (local address 127.0.0.1:34252)
0924 16:10:52.303759 (+ 5090us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:52.303785 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:52.304904 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:52.305683 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:52.305689 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:52.305727 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:10:52.306236 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:52.306249 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.308152 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.308158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:52.310368 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:52.310385 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.312381 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.312388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:52.312578 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:52.312614 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 16:10:52.312705 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 16:10:52.312845 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4924,"thread_start_us":4719,"threads_started":1}
I20260924 16:10:52.313282 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:52.298833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34252 (local address 127.3.214.195:32821)
0924 16:10:52.303229 (+ 4396us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:52.303239 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:52.305057 (+ 1818us) server_negotiation.cc:407] Connection header received
0924 16:10:52.305178 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:52.305187 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:52.305461 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:52.305681 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:10:52.306449 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.307964 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.310617 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.312152 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.312309 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:52.312586 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:52.312799 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 16:10:52.312952 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4201,"thread_start_us":4025,"threads_started":1}
I20260924 16:10:52.314348 22169 tablet_service.cc:1467] Processing CreateTablet for tablet 68fd5cf5ff0d458bb04964f07a542dd1 (DEFAULT_TABLE table=test-table [id=0f33a001ff624bcc8e738976e5eee99a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:10:52.315783 22169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68fd5cf5ff0d458bb04964f07a542dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:10:52.318815 22037 tablet_service.cc:1467] Processing CreateTablet for tablet 68fd5cf5ff0d458bb04964f07a542dd1 (DEFAULT_TABLE table=test-table [id=0f33a001ff624bcc8e738976e5eee99a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:10:52.318817 22302 tablet_service.cc:1467] Processing CreateTablet for tablet 68fd5cf5ff0d458bb04964f07a542dd1 (DEFAULT_TABLE table=test-table [id=0f33a001ff624bcc8e738976e5eee99a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:10:52.320897 22037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68fd5cf5ff0d458bb04964f07a542dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:10:52.320894 22302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68fd5cf5ff0d458bb04964f07a542dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:10:52.330283 22387 tablet_bootstrap.cc:492] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b: Bootstrap starting.
I20260924 16:10:52.333973 22387 tablet_bootstrap.cc:654] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b: Neither blocks nor log segments found. Creating new log.
I20260924 16:10:52.335249 22387 log.cc:824] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:10:52.341095 22387 tablet_bootstrap.cc:492] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b: No bootstrap required, opened a new log
I20260924 16:10:52.341409 22387 ts_tablet_manager.cc:1397] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260924 16:10:52.343072 22389 tablet_bootstrap.cc:492] T 68fd5cf5ff0d458bb04964f07a542dd1 P 21ce193f023c45ba997b76609635e23d: Bootstrap starting.
I20260924 16:10:52.343670 22390 tablet_bootstrap.cc:492] T 68fd5cf5ff0d458bb04964f07a542dd1 P 57697a03d0a54f2c9943b81cafc275f0: Bootstrap starting.
I20260924 16:10:52.347810 22389 tablet_bootstrap.cc:654] T 68fd5cf5ff0d458bb04964f07a542dd1 P 21ce193f023c45ba997b76609635e23d: Neither blocks nor log segments found. Creating new log.
I20260924 16:10:52.347815 22390 tablet_bootstrap.cc:654] T 68fd5cf5ff0d458bb04964f07a542dd1 P 57697a03d0a54f2c9943b81cafc275f0: Neither blocks nor log segments found. Creating new log.
I20260924 16:10:52.349382 22389 log.cc:824] T 68fd5cf5ff0d458bb04964f07a542dd1 P 21ce193f023c45ba997b76609635e23d: Log is configured to *not* fsync() on all Append() calls
I20260924 16:10:52.349380 22390 log.cc:824] T 68fd5cf5ff0d458bb04964f07a542dd1 P 57697a03d0a54f2c9943b81cafc275f0: Log is configured to *not* fsync() on all Append() calls
I20260924 16:10:52.349764 22387 raft_consensus.cc:348] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ce193f023c45ba997b76609635e23d" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43425 } } peers { permanent_uuid: "57697a03d0a54f2c9943b81cafc275f0" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 32821 } } peers { permanent_uuid: "9ed58339f0e442638f54286f7ff4226b" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45947 } }
I20260924 16:10:52.350265 22387 raft_consensus.cc:374] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:10:52.350368 22387 raft_consensus.cc:729] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed58339f0e442638f54286f7ff4226b, State: Initialized, Role: FOLLOWER
I20260924 16:10:52.351266 22387 consensus_queue.cc:260] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ce193f023c45ba997b76609635e23d" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43425 } } peers { permanent_uuid: "57697a03d0a54f2c9943b81cafc275f0" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 32821 } } peers { permanent_uuid: "9ed58339f0e442638f54286f7ff4226b" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45947 } }
I20260924 16:10:52.352859 22234 heartbeater.cc:503] Master 127.3.214.254:40895 was elected leader, sending a full tablet report...
I20260924 16:10:52.353569 22387 ts_tablet_manager.cc:1428] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260924 16:10:52.354266 22389 tablet_bootstrap.cc:492] T 68fd5cf5ff0d458bb04964f07a542dd1 P 21ce193f023c45ba997b76609635e23d: No bootstrap required, opened a new log
I20260924 16:10:52.354650 22389 ts_tablet_manager.cc:1397] T 68fd5cf5ff0d458bb04964f07a542dd1 P 21ce193f023c45ba997b76609635e23d: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260924 16:10:52.356235 22390 tablet_bootstrap.cc:492] T 68fd5cf5ff0d458bb04964f07a542dd1 P 57697a03d0a54f2c9943b81cafc275f0: No bootstrap required, opened a new log
I20260924 16:10:52.356592 22390 ts_tablet_manager.cc:1397] T 68fd5cf5ff0d458bb04964f07a542dd1 P 57697a03d0a54f2c9943b81cafc275f0: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20260924 16:10:52.364109 22389 raft_consensus.cc:348] T 68fd5cf5ff0d458bb04964f07a542dd1 P 21ce193f023c45ba997b76609635e23d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ce193f023c45ba997b76609635e23d" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43425 } } peers { permanent_uuid: "57697a03d0a54f2c9943b81cafc275f0" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 32821 } } peers { permanent_uuid: "9ed58339f0e442638f54286f7ff4226b" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45947 } }
I20260924 16:10:52.364619 22389 raft_consensus.cc:374] T 68fd5cf5ff0d458bb04964f07a542dd1 P 21ce193f023c45ba997b76609635e23d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:10:52.364722 22389 raft_consensus.cc:729] T 68fd5cf5ff0d458bb04964f07a542dd1 P 21ce193f023c45ba997b76609635e23d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21ce193f023c45ba997b76609635e23d, State: Initialized, Role: FOLLOWER
I20260924 16:10:52.365402 22389 consensus_queue.cc:260] T 68fd5cf5ff0d458bb04964f07a542dd1 P 21ce193f023c45ba997b76609635e23d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ce193f023c45ba997b76609635e23d" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43425 } } peers { permanent_uuid: "57697a03d0a54f2c9943b81cafc275f0" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 32821 } } peers { permanent_uuid: "9ed58339f0e442638f54286f7ff4226b" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45947 } }
I20260924 16:10:52.365861 22390 raft_consensus.cc:348] T 68fd5cf5ff0d458bb04964f07a542dd1 P 57697a03d0a54f2c9943b81cafc275f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ce193f023c45ba997b76609635e23d" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43425 } } peers { permanent_uuid: "57697a03d0a54f2c9943b81cafc275f0" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 32821 } } peers { permanent_uuid: "9ed58339f0e442638f54286f7ff4226b" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45947 } }
I20260924 16:10:52.366504 22390 raft_consensus.cc:374] T 68fd5cf5ff0d458bb04964f07a542dd1 P 57697a03d0a54f2c9943b81cafc275f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:10:52.366624 22390 raft_consensus.cc:729] T 68fd5cf5ff0d458bb04964f07a542dd1 P 57697a03d0a54f2c9943b81cafc275f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57697a03d0a54f2c9943b81cafc275f0, State: Initialized, Role: FOLLOWER
I20260924 16:10:52.367270 22390 consensus_queue.cc:260] T 68fd5cf5ff0d458bb04964f07a542dd1 P 57697a03d0a54f2c9943b81cafc275f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ce193f023c45ba997b76609635e23d" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43425 } } peers { permanent_uuid: "57697a03d0a54f2c9943b81cafc275f0" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 32821 } } peers { permanent_uuid: "9ed58339f0e442638f54286f7ff4226b" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45947 } }
I20260924 16:10:52.367556 22389 ts_tablet_manager.cc:1428] T 68fd5cf5ff0d458bb04964f07a542dd1 P 21ce193f023c45ba997b76609635e23d: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260924 16:10:52.368410 22367 heartbeater.cc:503] Master 127.3.214.254:40895 was elected leader, sending a full tablet report...
I20260924 16:10:52.369170 22390 ts_tablet_manager.cc:1428] T 68fd5cf5ff0d458bb04964f07a542dd1 P 57697a03d0a54f2c9943b81cafc275f0: Time spent starting tablet: real 0.012s user 0.010s sys 0.003s
W20260924 16:10:52.392866 22235 tablet.cc:2369] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:10:52.475636 22368 tablet.cc:2369] T 68fd5cf5ff0d458bb04964f07a542dd1 P 57697a03d0a54f2c9943b81cafc275f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:10:52.592212 22393 raft_consensus.cc:482] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:10:52.592595 22393 raft_consensus.cc:504] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ce193f023c45ba997b76609635e23d" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43425 } } peers { permanent_uuid: "57697a03d0a54f2c9943b81cafc275f0" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 32821 } } peers { permanent_uuid: "9ed58339f0e442638f54286f7ff4226b" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45947 } }
I20260924 16:10:52.594918 22393 leader_election.cc:288] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21ce193f023c45ba997b76609635e23d (127.3.214.193:43425), 57697a03d0a54f2c9943b81cafc275f0 (127.3.214.195:32821)
W20260924 16:10:52.602393 22103 tablet.cc:2369] T 68fd5cf5ff0d458bb04964f07a542dd1 P 21ce193f023c45ba997b76609635e23d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:10:52.604986 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:52.595304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:43425 (local address 127.3.214.194:44209)
0924 16:10:52.596353 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:52.596387 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:52.596824 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:52.597509 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:52.597515 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:52.597552 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:10:52.597981 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:52.597992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.599789 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.599797 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:52.602357 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:52.602387 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.604215 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.604222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:52.604410 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:52.604441 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 16:10:52.604525 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 16:10:52.604675 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"thread_start_us":523,"threads_started":1}
I20260924 16:10:52.605060 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:52.595976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:44209 (local address 127.3.214.193:43425)
0924 16:10:52.596206 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:52.596212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:52.596549 (+ 337us) server_negotiation.cc:407] Connection header received
0924 16:10:52.597158 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:52.597167 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:52.597296 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:52.597443 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:10:52.598190 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.599513 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.602694 (+ 3181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.603972 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.604118 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:52.604178 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:52.604598 (+ 420us) server_negotiation.cc:299] Negotiation successful
0924 16:10:52.604758 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 16:10:52.606504 22057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68fd5cf5ff0d458bb04964f07a542dd1" candidate_uuid: "9ed58339f0e442638f54286f7ff4226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ce193f023c45ba997b76609635e23d" is_pre_election: true
I20260924 16:10:52.607223 22057 raft_consensus.cc:2454] T 68fd5cf5ff0d458bb04964f07a542dd1 P 21ce193f023c45ba997b76609635e23d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ed58339f0e442638f54286f7ff4226b in term 0.
I20260924 16:10:52.608246 22122 leader_election.cc:302] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21ce193f023c45ba997b76609635e23d, 9ed58339f0e442638f54286f7ff4226b; no voters:
I20260924 16:10:52.609022 22393 raft_consensus.cc:2781] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:10:52.609259 22393 raft_consensus.cc:482] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:10:52.609404 22393 raft_consensus.cc:3037] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:10:52.610349 22397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:52.596405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:32821 (local address 127.3.214.194:49403)
0924 16:10:52.602920 (+ 6515us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:52.602957 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:52.603183 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:52.603635 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:52.603638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:52.603665 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:10:52.604067 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:52.604081 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.605482 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.605490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:52.607517 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:52.607543 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.609433 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.609442 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:52.609652 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:52.609688 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 16:10:52.609804 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 16:10:52.609974 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6322,"thread_start_us":5986,"threads_started":1}
I20260924 16:10:52.610374 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:52.596650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:49403 (local address 127.3.214.195:32821)
0924 16:10:52.596887 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:52.596894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:52.603133 (+ 6239us) server_negotiation.cc:407] Connection header received
0924 16:10:52.603357 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:52.603363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:52.603472 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:52.603594 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:10:52.604289 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.605235 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.607895 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.609128 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.609369 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:52.609487 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:52.609871 (+ 384us) server_negotiation.cc:299] Negotiation successful
0924 16:10:52.610038 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 16:10:52.611721 22322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68fd5cf5ff0d458bb04964f07a542dd1" candidate_uuid: "9ed58339f0e442638f54286f7ff4226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57697a03d0a54f2c9943b81cafc275f0" is_pre_election: true
I20260924 16:10:52.612298 22322 raft_consensus.cc:2454] T 68fd5cf5ff0d458bb04964f07a542dd1 P 57697a03d0a54f2c9943b81cafc275f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ed58339f0e442638f54286f7ff4226b in term 0.
I20260924 16:10:52.612718 22393 raft_consensus.cc:504] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ce193f023c45ba997b76609635e23d" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43425 } } peers { permanent_uuid: "57697a03d0a54f2c9943b81cafc275f0" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 32821 } } peers { permanent_uuid: "9ed58339f0e442638f54286f7ff4226b" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45947 } }
I20260924 16:10:52.613557 22393 leader_election.cc:288] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b [CANDIDATE]: Term 1 election: Requested vote from peers 21ce193f023c45ba997b76609635e23d (127.3.214.193:43425), 57697a03d0a54f2c9943b81cafc275f0 (127.3.214.195:32821)
I20260924 16:10:52.614235 22322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68fd5cf5ff0d458bb04964f07a542dd1" candidate_uuid: "9ed58339f0e442638f54286f7ff4226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57697a03d0a54f2c9943b81cafc275f0"
I20260924 16:10:52.614193 22057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68fd5cf5ff0d458bb04964f07a542dd1" candidate_uuid: "9ed58339f0e442638f54286f7ff4226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ce193f023c45ba997b76609635e23d"
I20260924 16:10:52.614495 22322 raft_consensus.cc:3037] T 68fd5cf5ff0d458bb04964f07a542dd1 P 57697a03d0a54f2c9943b81cafc275f0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:10:52.614496 22057 raft_consensus.cc:3037] T 68fd5cf5ff0d458bb04964f07a542dd1 P 21ce193f023c45ba997b76609635e23d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:10:52.618000 22057 raft_consensus.cc:2454] T 68fd5cf5ff0d458bb04964f07a542dd1 P 21ce193f023c45ba997b76609635e23d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed58339f0e442638f54286f7ff4226b in term 1.
I20260924 16:10:52.618001 22322 raft_consensus.cc:2454] T 68fd5cf5ff0d458bb04964f07a542dd1 P 57697a03d0a54f2c9943b81cafc275f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed58339f0e442638f54286f7ff4226b in term 1.
I20260924 16:10:52.618660 22122 leader_election.cc:302] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57697a03d0a54f2c9943b81cafc275f0, 9ed58339f0e442638f54286f7ff4226b; no voters:
I20260924 16:10:52.619088 22393 raft_consensus.cc:2781] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:10:52.619693 22393 raft_consensus.cc:686] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b [term 1 LEADER]: Becoming Leader. State: Replica: 9ed58339f0e442638f54286f7ff4226b, State: Running, Role: LEADER
I20260924 16:10:52.620397 22393 consensus_queue.cc:237] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ce193f023c45ba997b76609635e23d" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43425 } } peers { permanent_uuid: "57697a03d0a54f2c9943b81cafc275f0" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 32821 } } peers { permanent_uuid: "9ed58339f0e442638f54286f7ff4226b" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45947 } }
I20260924 16:10:52.626194 21915 catalog_manager.cc:5495] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ed58339f0e442638f54286f7ff4226b (127.3.214.194). New cstate: current_term: 1 leader_uuid: "9ed58339f0e442638f54286f7ff4226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ce193f023c45ba997b76609635e23d" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57697a03d0a54f2c9943b81cafc275f0" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 32821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed58339f0e442638f54286f7ff4226b" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45947 } health_report { overall_health: HEALTHY } } }
I20260924 16:10:52.695060 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:52.688259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40536 (local address 127.3.214.194:45947)
0924 16:10:52.688536 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:52.688542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:52.688568 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:10:52.688913 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:52.688919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:52.689053 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:52.689250 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:10:52.689635 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.690488 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.692188 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.692720 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.692852 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:52.693497 (+ 645us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:10:52.694333 (+ 836us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:10:52.694463 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:52.694589 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 16:10:52.694721 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 16:10:52.706660 22057 raft_consensus.cc:1261] T 68fd5cf5ff0d458bb04964f07a542dd1 P 21ce193f023c45ba997b76609635e23d [term 1 FOLLOWER]: Refusing update from remote peer 9ed58339f0e442638f54286f7ff4226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:10:52.706660 22322 raft_consensus.cc:1261] T 68fd5cf5ff0d458bb04964f07a542dd1 P 57697a03d0a54f2c9943b81cafc275f0 [term 1 FOLLOWER]: Refusing update from remote peer 9ed58339f0e442638f54286f7ff4226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:10:52.708190 22398 consensus_queue.cc:1035] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "57697a03d0a54f2c9943b81cafc275f0" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 32821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:10:52.708652 22393 consensus_queue.cc:1035] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "21ce193f023c45ba997b76609635e23d" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:10:52.725579 22402 mvcc.cc:205] Tried to move back new op lower bound from 7332930571069796352 to 7332930570737729536. Current Snapshot: MvccSnapshot[applied={T|T < 7332930571069796352}]
I20260924 16:10:52.742183 21914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:10:52.742491 21914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:10:52.747246 21914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74313c9778cc4e35a99eba644f45bb58: Sending DeleteTablet for 3 replicas of tablet 68fd5cf5ff0d458bb04964f07a542dd1
I20260924 16:10:52.748787 22037 tablet_service.cc:1514] Processing DeleteTablet for tablet 68fd5cf5ff0d458bb04964f07a542dd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:10:52 UTC) from {username='slave'} at 127.0.0.1:52758
I20260924 16:10:52.748793 22302 tablet_service.cc:1514] Processing DeleteTablet for tablet 68fd5cf5ff0d458bb04964f07a542dd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:10:52 UTC) from {username='slave'} at 127.0.0.1:34252
I20260924 16:10:52.749668 22169 tablet_service.cc:1514] Processing DeleteTablet for tablet 68fd5cf5ff0d458bb04964f07a542dd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:10:52 UTC) from {username='slave'} at 127.0.0.1:40522
I20260924 16:10:52.750381 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 21974
I20260924 16:10:52.764210 22411 tablet_replica.cc:330] T 68fd5cf5ff0d458bb04964f07a542dd1 P 57697a03d0a54f2c9943b81cafc275f0: stopping tablet replica
I20260924 16:10:52.764858 22411 raft_consensus.cc:2229] T 68fd5cf5ff0d458bb04964f07a542dd1 P 57697a03d0a54f2c9943b81cafc275f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:10:52.765515 22411 raft_consensus.cc:2258] T 68fd5cf5ff0d458bb04964f07a542dd1 P 57697a03d0a54f2c9943b81cafc275f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:10:52.766531 22412 tablet_replica.cc:330] T 68fd5cf5ff0d458bb04964f07a542dd1 P 21ce193f023c45ba997b76609635e23d: stopping tablet replica
I20260924 16:10:52.766985 22412 raft_consensus.cc:2229] T 68fd5cf5ff0d458bb04964f07a542dd1 P 21ce193f023c45ba997b76609635e23d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:10:52.767066 22413 tablet_replica.cc:330] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b: stopping tablet replica
I20260924 16:10:52.767525 22412 raft_consensus.cc:2258] T 68fd5cf5ff0d458bb04964f07a542dd1 P 21ce193f023c45ba997b76609635e23d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:10:52.767459 22413 raft_consensus.cc:2229] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:10:52.768257 22413 raft_consensus.cc:2258] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:10:52.770769 22412 ts_tablet_manager.cc:1905] T 68fd5cf5ff0d458bb04964f07a542dd1 P 21ce193f023c45ba997b76609635e23d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:10:52.770776 22413 ts_tablet_manager.cc:1905] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:10:52.774065 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:52.752112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52762 (local address 127.3.214.193:43425)
0924 16:10:52.752346 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:52.752354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:52.763252 (+ 10898us) server_negotiation.cc:407] Connection header received
0924 16:10:52.763885 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:52.763894 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:52.764064 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:52.764167 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:52.765224 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.766796 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.769144 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.769745 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.769871 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:52.771813 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:52.771827 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:52.771857 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:10:52.772923 (+ 1066us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:52.773005 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:10:52.773044 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:52.773149 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:52.773433 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 16:10:52.773585 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 16:10:52.775933 22097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:10:52.777523 22411 ts_tablet_manager.cc:1905] T 68fd5cf5ff0d458bb04964f07a542dd1 P 57697a03d0a54f2c9943b81cafc275f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:10:52.778753 22413 ts_tablet_manager.cc:1918] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:10:52.778991 22413 log.cc:1196] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/68fd5cf5ff0d458bb04964f07a542dd1
I20260924 16:10:52.779663 22413 ts_tablet_manager.cc:1939] T 68fd5cf5ff0d458bb04964f07a542dd1 P 9ed58339f0e442638f54286f7ff4226b: Deleting consensus metadata
I20260924 16:10:52.782200 21899 catalog_manager.cc:4841] TS 9ed58339f0e442638f54286f7ff4226b (127.3.214.194:45947): tablet 68fd5cf5ff0d458bb04964f07a542dd1 (table test-table [id=0f33a001ff624bcc8e738976e5eee99a]) successfully deleted
I20260924 16:10:52.786033 22411 ts_tablet_manager.cc:1918] T 68fd5cf5ff0d458bb04964f07a542dd1 P 57697a03d0a54f2c9943b81cafc275f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:10:52.786376 22411 log.cc:1196] T 68fd5cf5ff0d458bb04964f07a542dd1 P 57697a03d0a54f2c9943b81cafc275f0: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/68fd5cf5ff0d458bb04964f07a542dd1
I20260924 16:10:52.787003 22411 ts_tablet_manager.cc:1939] T 68fd5cf5ff0d458bb04964f07a542dd1 P 57697a03d0a54f2c9943b81cafc275f0: Deleting consensus metadata
I20260924 16:10:52.789383 21900 catalog_manager.cc:4841] TS 57697a03d0a54f2c9943b81cafc275f0 (127.3.214.195:32821): tablet 68fd5cf5ff0d458bb04964f07a542dd1 (table test-table [id=0f33a001ff624bcc8e738976e5eee99a]) successfully deleted
I20260924 16:10:52.881166 22412 ts_tablet_manager.cc:1918] T 68fd5cf5ff0d458bb04964f07a542dd1 P 21ce193f023c45ba997b76609635e23d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:10:52.881443 22412 log.cc:1196] T 68fd5cf5ff0d458bb04964f07a542dd1 P 21ce193f023c45ba997b76609635e23d: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790265920616583-3931-0/minicluster-data/ts-0/wal/wals/68fd5cf5ff0d458bb04964f07a542dd1
I20260924 16:10:52.882094 22412 ts_tablet_manager.cc:1939] T 68fd5cf5ff0d458bb04964f07a542dd1 P 21ce193f023c45ba997b76609635e23d: Deleting consensus metadata
I20260924 16:10:52.883548 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 21974
W20260924 16:10:52.901577 21900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.214.193:43425 (error 108)
I20260924 16:10:52.901906 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 22106
I20260924 16:10:52.904347 22417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:52.903619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:43425 (local address 127.0.0.1:52778)
0924 16:10:52.904057 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:52.904174 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:43425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":221,"threads_started":1}
W20260924 16:10:52.904850 21900 catalog_manager.cc:4584] TS 21ce193f023c45ba997b76609635e23d (127.3.214.193:43425): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 68fd5cf5ff0d458bb04964f07a542dd1: Network error: Client connection negotiation failed: client connection to 127.3.214.193:43425: connect: Connection refused (error 111)
I20260924 16:10:52.912038 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:52.903138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40548 (local address 127.3.214.194:45947)
0924 16:10:52.903349 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:52.903356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:52.904042 (+ 686us) server_negotiation.cc:407] Connection header received
0924 16:10:52.904136 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:52.904142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:52.904268 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:52.904403 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:52.904973 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.906003 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.908361 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:52.908882 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:52.909017 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:52.910346 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:52.910364 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:52.910387 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:10:52.911139 (+ 752us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:52.911201 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:10:52.911224 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:52.911330 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:52.911490 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 16:10:52.911603 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:10:52.912951 22229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:10:52.958837 22417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:52.958362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:43425 (local address 127.0.0.1:52792)
0924 16:10:52.958592 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:52.958696 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:43425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 16:10:52.960453 22417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:52.959902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:43425 (local address 127.0.0.1:52794)
0924 16:10:52.960157 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:52.960285 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:43425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260924 16:10:53.021515 22417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:53.020856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:43425 (local address 127.0.0.1:52796)
0924 16:10:53.021172 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:53.021304 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:43425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260924 16:10:53.023037 22417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:53.022532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:43425 (local address 127.0.0.1:52804)
0924 16:10:53.022749 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:53.022854 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:43425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 16:10:53.045024 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 22106
I20260924 16:10:53.066633 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 22238
I20260924 16:10:53.075178 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:53.067828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.3.214.195:32821)
0924 16:10:53.068047 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:53.068052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:53.068075 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:10:53.068401 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:53.068408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:53.068543 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:53.068670 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:53.069235 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:53.070020 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:53.071617 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:53.072052 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:53.072165 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:53.073439 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:53.073448 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:53.073469 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:10:53.074137 (+ 668us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:53.074210 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:10:53.074236 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:53.074337 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:53.074657 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 16:10:53.074782 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 16:10:53.076125 22362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:10:53.104200 22417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:53.103606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:43425 (local address 127.0.0.1:52816)
0924 16:10:53.103868 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:53.103975 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:43425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260924 16:10:53.105724 22417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:53.105202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:43425 (local address 127.0.0.1:52824)
0924 16:10:53.105442 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:53.105531 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:43425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260924 16:10:53.171780 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 22238
I20260924 16:10:53.189054 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 21884
I20260924 16:10:53.190620 21945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:10:53.308173 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 21884
2026-09-24T16:10:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3404 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:10:53 dist-test-slave-1gc3 krb5kdc[22423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:10:53 dist-test-slave-1gc3 krb5kdc[22423](info): set up 2 sockets
Sep 24 16:10:53 dist-test-slave-1gc3 krb5kdc[22423](info): commencing operation
krb5kdc: starting...
W20260924 16:10:54.383857 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:10:54 dist-test-slave-1gc3 krb5kdc[22423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:10:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:10:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:10:54.705924 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:45515
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:45515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:10:55.128141 22436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:10:55.128603 22436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:10:55.128756 22436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:10:55.137135 22436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:10:55.137331 22436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:10:55.137422 22436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:10:55.137514 22436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:10:55.137583 22436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:10:55.150785 22436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:45515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:45515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:10:55.152678 22436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:10:55.154506 22436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:10:55.169687 22444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:55.169781 22442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:55.169806 22441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:55.170977 22436 server_base.cc:1034] running on GCE node
I20260924 16:10:55.172014 22436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:10:55.174010 22436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:10:55.175285 22436 hybrid_clock.cc:648] HybridClock initialized: now 1790266255175219 us; error 51 us; skew 500 ppm
I20260924 16:10:55.175737 22436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:10:55 dist-test-slave-1gc3 krb5kdc[22423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266255, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:10:55.179510 22436 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:10:55.181458 22436 webserver.cc:460] Webserver started at http://127.3.214.254:34137/ using document root <none> and password file <none>
I20260924 16:10:55.182427 22436 fs_manager.cc:359] Metadata directory not provided
I20260924 16:10:55.182570 22436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:10:55.182960 22436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:10:55.186952 22436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "63243771ca1f43c4ade500a8732c0d8b"
format_stamp: "Formatted at 2026-09-24 16:10:55 on dist-test-slave-1gc3"
server_key: "610a31fab8d67d709c2d63298a6a1999"
server_key_iv: "0ac0f5412c0bf5a939f91743d7cfcd74"
server_key_version: "encryptionkey@0"
I20260924 16:10:55.187960 22436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "63243771ca1f43c4ade500a8732c0d8b"
format_stamp: "Formatted at 2026-09-24 16:10:55 on dist-test-slave-1gc3"
server_key: "610a31fab8d67d709c2d63298a6a1999"
server_key_iv: "0ac0f5412c0bf5a939f91743d7cfcd74"
server_key_version: "encryptionkey@0"
I20260924 16:10:55.195094 22436 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260924 16:10:55.199668 22451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:55.201416 22436 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260924 16:10:55.201624 22436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "63243771ca1f43c4ade500a8732c0d8b"
format_stamp: "Formatted at 2026-09-24 16:10:55 on dist-test-slave-1gc3"
server_key: "610a31fab8d67d709c2d63298a6a1999"
server_key_iv: "0ac0f5412c0bf5a939f91743d7cfcd74"
server_key_version: "encryptionkey@0"
I20260924 16:10:55.201855 22436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:10:55.224840 22436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:10:55.230520 22436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:10:55.230872 22436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:10:55.253614 22436 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:45515
I20260924 16:10:55.253623 22502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:45515 every 8 connection(s)
I20260924 16:10:55.255424 22436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:10:55.259965 22503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:10:55.260519 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 22436
I20260924 16:10:55.260730 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:10:55.261303 3931 external_mini_cluster.cc:1428] Setting key 4b201bd092fc575ab6074903a04033b3
I20260924 16:10:55.276582 22503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b: Bootstrap starting.
Sep 24 16:10:55 dist-test-slave-1gc3 krb5kdc[22423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:10:55.281068 22503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b: Neither blocks nor log segments found. Creating new log.
I20260924 16:10:55.282457 22503 log.cc:824] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:10:55.286728 22503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b: No bootstrap required, opened a new log
I20260924 16:10:55.291689 22506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:55.264005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37738 (local address 127.3.214.254:45515)
0924 16:10:55.265103 (+ 1098us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:55.265124 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:55.265187 (+ 63us) server_negotiation.cc:407] Connection header received
0924 16:10:55.267438 (+ 2251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:55.267481 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:55.268121 (+ 640us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:55.268609 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:55.270716 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:55.272314 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:55.274397 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:55.275328 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:55.275513 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:55.281318 (+ 5805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:55.281332 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:55.281361 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:55.281439 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:55.285526 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:55.286881 (+ 1355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:55.286894 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:55.286907 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:55.287238 (+ 331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:55.287762 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:55.287765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:55.287768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:55.288571 (+ 803us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:55.288907 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:55.289500 (+ 593us) server_negotiation.cc:299] Negotiation successful
0924 16:10:55.290191 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":812,"thread_start_us":466,"threads_started":1}
I20260924 16:10:55.296181 22503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63243771ca1f43c4ade500a8732c0d8b" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 45515 } }
I20260924 16:10:55.296639 22503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:10:55.296748 22503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63243771ca1f43c4ade500a8732c0d8b, State: Initialized, Role: FOLLOWER
I20260924 16:10:55.297372 22503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63243771ca1f43c4ade500a8732c0d8b" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 45515 } }
I20260924 16:10:55.297629 22503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:10:55.297770 22503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:10:55.297940 22503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:10:55.300518 22503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63243771ca1f43c4ade500a8732c0d8b" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 45515 } }
I20260924 16:10:55.301143 22503 leader_election.cc:302] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63243771ca1f43c4ade500a8732c0d8b; no voters:
I20260924 16:10:55.301891 22503 leader_election.cc:288] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:10:55.302093 22508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:10:55.303109 22508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b [term 1 LEADER]: Becoming Leader. State: Replica: 63243771ca1f43c4ade500a8732c0d8b, State: Running, Role: LEADER
I20260924 16:10:55.303867 22508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63243771ca1f43c4ade500a8732c0d8b" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 45515 } }
I20260924 16:10:55.304430 22503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:10:55.307848 22510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63243771ca1f43c4ade500a8732c0d8b. Latest consensus state: current_term: 1 leader_uuid: "63243771ca1f43c4ade500a8732c0d8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63243771ca1f43c4ade500a8732c0d8b" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 45515 } } }
I20260924 16:10:55.308671 22510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b [sys.catalog]: This master's current role is: LEADER
I20260924 16:10:55.307919 22509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63243771ca1f43c4ade500a8732c0d8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63243771ca1f43c4ade500a8732c0d8b" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 45515 } } }
I20260924 16:10:55.308833 22509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b [sys.catalog]: This master's current role is: LEADER
I20260924 16:10:55.311081 22516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:10:55.319404 22516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:10:55.331961 22516 catalog_manager.cc:1347] Generated new cluster ID: 088ce26ab56145c99ed52fafa9a0d67b
I20260924 16:10:55.332111 22516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:10:55.352902 22516 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:10:55.355417 22516 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:10:55.367302 22516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b: Generated new TSK 0
I20260924 16:10:55.368829 22516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.214.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
I20260924 16:10:55.445461 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:45515
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:10:55.828722 22529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:10:55.829044 22529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:10:55.829162 22529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:10:55.836773 22529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:10:55.836923 22529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:10:55.837080 22529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:10:55.847981 22529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:45515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:10:55.849524 22529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:10:55.851414 22529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:10:55.868079 22535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:55.868127 22534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:55.868296 22537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:55.869123 22529 server_base.cc:1034] running on GCE node
I20260924 16:10:55.870029 22529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:10:55.871488 22529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:10:55.872794 22529 hybrid_clock.cc:648] HybridClock initialized: now 1790266255872702 us; error 68 us; skew 500 ppm
I20260924 16:10:55.873266 22529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:10:55 dist-test-slave-1gc3 krb5kdc[22423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266255, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:10:55.881534 22529 init.cc:377] Logged in from keytab as kudu/127.3.214.193@KRBTEST.COM (short username kudu)
I20260924 16:10:55.883823 22529 webserver.cc:460] Webserver started at http://127.3.214.193:40847/ using document root <none> and password file <none>
I20260924 16:10:55.884785 22529 fs_manager.cc:359] Metadata directory not provided
I20260924 16:10:55.884925 22529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:10:55.885331 22529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:10:55.889273 22529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "3323ef1ac8254f42a936ec0c7239fe09"
format_stamp: "Formatted at 2026-09-24 16:10:55 on dist-test-slave-1gc3"
server_key: "f51dbf84924903d99fd89ba04a461ebb"
server_key_iv: "073e249adab52e421ae9932ef8ced1f4"
server_key_version: "encryptionkey@0"
I20260924 16:10:55.890383 22529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "3323ef1ac8254f42a936ec0c7239fe09"
format_stamp: "Formatted at 2026-09-24 16:10:55 on dist-test-slave-1gc3"
server_key: "f51dbf84924903d99fd89ba04a461ebb"
server_key_iv: "073e249adab52e421ae9932ef8ced1f4"
server_key_version: "encryptionkey@0"
I20260924 16:10:55.897609 22529 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260924 16:10:55.902827 22544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:55.904929 22529 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:10:55.905249 22529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "3323ef1ac8254f42a936ec0c7239fe09"
format_stamp: "Formatted at 2026-09-24 16:10:55 on dist-test-slave-1gc3"
server_key: "f51dbf84924903d99fd89ba04a461ebb"
server_key_iv: "073e249adab52e421ae9932ef8ced1f4"
server_key_version: "encryptionkey@0"
I20260924 16:10:55.905546 22529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:10:55.930308 22529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:10:55.936411 22529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:10:55.936772 22529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:10:55.938164 22529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:10:55.940007 22529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:10:55.940096 22529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:55.940223 22529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:10:55.940291 22529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:55.978003 22529 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:44591
I20260924 16:10:55.978053 22657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:44591 every 8 connection(s)
I20260924 16:10:55.979707 22529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:10:55.984871 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 22529
I20260924 16:10:55.985083 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:10:55.985598 3931 external_mini_cluster.cc:1428] Setting key df3795aeb86329f3b5f2b18a606c3491
Sep 24 16:10:55 dist-test-slave-1gc3 krb5kdc[22423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266255, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:10:56.004479 22660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:55.982987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:40003 (local address 127.3.214.254:45515)
0924 16:10:55.983585 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:55.983591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:55.984883 (+ 1292us) server_negotiation.cc:407] Connection header received
0924 16:10:55.987528 (+ 2645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:55.987533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:55.987620 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:55.987747 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:55.989736 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:55.990588 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:55.992281 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:55.992721 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:55.992850 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:55.996895 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:55.996901 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:55.996905 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:55.996948 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:56.000203 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:56.001494 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:56.001500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:56.001504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:56.001709 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:56.002282 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:56.002286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:56.002289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:56.002689 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:56.002868 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:56.003821 (+ 953us) server_negotiation.cc:299] Negotiation successful
0924 16:10:56.004104 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":296,"threads_started":1}
I20260924 16:10:56.005863 22661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:55.983664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:45515 (local address 127.3.214.193:40003)
0924 16:10:55.984657 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:55.984722 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:55.987134 (+ 2412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:55.988113 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:55.988126 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:55.988803 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:10:55.989505 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:55.989526 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:55.990757 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:55.990761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:55.992109 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:55.992125 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:55.993003 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:55.993011 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:55.993260 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:55.994145 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:10:55.994192 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:10:55.996604 (+ 2412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:10:56.000520 (+ 3916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:10:56.000533 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:10:56.000558 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:10:56.001302 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:10:56.001841 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:10:56.001844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:10:56.001847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:10:56.002156 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:10:56.002833 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:10:56.002841 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:10:56.003357 (+ 516us) client_negotiation.cc:769] Sending connection context
0924 16:10:56.003653 (+ 296us) client_negotiation.cc:240] Negotiation successful
0924 16:10:56.004329 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":366,"threads_started":1}
I20260924 16:10:56.008104 22658 heartbeater.cc:347] Connected to a master server at 127.3.214.254:45515
I20260924 16:10:56.008484 22658 heartbeater.cc:464] Registering TS with master...
I20260924 16:10:56.009404 22658 heartbeater.cc:511] Master 127.3.214.254:45515 requested a full tablet report, sending...
I20260924 16:10:56.012668 22468 ts_manager.cc:194] Registered new tserver with Master: 3323ef1ac8254f42a936ec0c7239fe09 (127.3.214.193:44591)
I20260924 16:10:56.014674 22468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:40003
WARNING: no policy specified for kudu/127.3.214.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
I20260924 16:10:56.053973 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:45515
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:10:56.413132 22664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:10:56.413481 22664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:10:56.413619 22664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:10:56.421109 22664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:10:56.421244 22664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:10:56.421362 22664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:10:56.432163 22664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:45515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:10:56.433583 22664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:10:56.435078 22664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:10:56.450528 22669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:56.450562 22670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:56.450606 22672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:56.451525 22664 server_base.cc:1034] running on GCE node
I20260924 16:10:56.452450 22664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:10:56.453657 22664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:10:56.455056 22664 hybrid_clock.cc:648] HybridClock initialized: now 1790266256454925 us; error 119 us; skew 500 ppm
I20260924 16:10:56.455488 22664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:10:56 dist-test-slave-1gc3 krb5kdc[22423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266256, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:10:56.459416 22664 init.cc:377] Logged in from keytab as kudu/127.3.214.194@KRBTEST.COM (short username kudu)
I20260924 16:10:56.461419 22664 webserver.cc:460] Webserver started at http://127.3.214.194:33155/ using document root <none> and password file <none>
I20260924 16:10:56.462284 22664 fs_manager.cc:359] Metadata directory not provided
I20260924 16:10:56.462430 22664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:10:56.462769 22664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:10:56.466089 22664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "2817261e09c94665a4a44684864d88c6"
format_stamp: "Formatted at 2026-09-24 16:10:56 on dist-test-slave-1gc3"
server_key: "81b1c8ed261e0bbd19323b333d3ff4fb"
server_key_iv: "7da67fb2469c6ae04e70723a2cf0c86f"
server_key_version: "encryptionkey@0"
I20260924 16:10:56.467052 22664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "2817261e09c94665a4a44684864d88c6"
format_stamp: "Formatted at 2026-09-24 16:10:56 on dist-test-slave-1gc3"
server_key: "81b1c8ed261e0bbd19323b333d3ff4fb"
server_key_iv: "7da67fb2469c6ae04e70723a2cf0c86f"
server_key_version: "encryptionkey@0"
I20260924 16:10:56.473444 22664 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 16:10:56.478176 22679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:56.480116 22664 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:10:56.480367 22664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "2817261e09c94665a4a44684864d88c6"
format_stamp: "Formatted at 2026-09-24 16:10:56 on dist-test-slave-1gc3"
server_key: "81b1c8ed261e0bbd19323b333d3ff4fb"
server_key_iv: "7da67fb2469c6ae04e70723a2cf0c86f"
server_key_version: "encryptionkey@0"
I20260924 16:10:56.480613 22664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:10:56.512259 22664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:10:56.517095 22664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:10:56.517390 22664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:10:56.518579 22664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:10:56.520205 22664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:10:56.520290 22664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:56.520418 22664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:10:56.520473 22664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:56.556120 22664 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:45523
I20260924 16:10:56.556154 22792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:45523 every 8 connection(s)
I20260924 16:10:56.557538 22664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:10:56.567041 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 22664
I20260924 16:10:56.567291 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:10:56.567924 3931 external_mini_cluster.cc:1428] Setting key ab9be2c70c342197331811191715ded1
Sep 24 16:10:56 dist-test-slave-1gc3 krb5kdc[22423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266256, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:10:56.584497 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:56.560826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:43993 (local address 127.3.214.254:45515)
0924 16:10:56.561365 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:56.561371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:56.562437 (+ 1066us) server_negotiation.cc:407] Connection header received
0924 16:10:56.564737 (+ 2300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:56.564744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:56.564847 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:56.564961 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:56.567374 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:56.568579 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:56.571041 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:56.571823 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:56.572005 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:56.576618 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:56.576628 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:56.576633 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:56.576689 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:56.580069 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:56.581331 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:56.581338 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:56.581342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:56.581552 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:56.582178 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:56.582183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:56.582186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:56.582571 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:56.582729 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:56.583803 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0924 16:10:56.584092 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":267,"threads_started":1}
I20260924 16:10:56.585925 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:56.561294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:45515 (local address 127.3.214.194:43993)
0924 16:10:56.562233 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:56.562291 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:56.564384 (+ 2093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:56.565298 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:56.565315 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:56.566057 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:10:56.567037 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:56.567063 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:56.568991 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:56.568998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:56.570802 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:56.570822 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:56.572156 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:56.572162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:56.572466 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:56.573353 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:10:56.573384 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:10:56.576170 (+ 2786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:10:56.580312 (+ 4142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:10:56.580324 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:10:56.580351 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:10:56.581103 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:10:56.581722 (+ 619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:10:56.581728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:10:56.581730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:10:56.581993 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:10:56.582768 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:10:56.582778 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:10:56.583335 (+ 557us) client_negotiation.cc:769] Sending connection context
0924 16:10:56.583659 (+ 324us) client_negotiation.cc:240] Negotiation successful
0924 16:10:56.584247 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":322,"threads_started":1}
I20260924 16:10:56.587865 22793 heartbeater.cc:347] Connected to a master server at 127.3.214.254:45515
I20260924 16:10:56.588229 22793 heartbeater.cc:464] Registering TS with master...
I20260924 16:10:56.589112 22793 heartbeater.cc:511] Master 127.3.214.254:45515 requested a full tablet report, sending...
I20260924 16:10:56.591158 22468 ts_manager.cc:194] Registered new tserver with Master: 2817261e09c94665a4a44684864d88c6 (127.3.214.194:45523)
I20260924 16:10:56.592262 22468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:43993
WARNING: no policy specified for kudu/127.3.214.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
I20260924 16:10:56.633657 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:45515
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:10:57.019166 22658 heartbeater.cc:503] Master 127.3.214.254:45515 was elected leader, sending a full tablet report...
W20260924 16:10:57.049914 22799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:10:57.050372 22799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:10:57.050540 22799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:10:57.058907 22799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:10:57.059083 22799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:10:57.059260 22799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:10:57.071071 22799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:45515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:10:57.072960 22799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:10:57.074777 22799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:10:57.092139 22807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:57.092206 22805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:57.092214 22804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:57.093247 22799 server_base.cc:1034] running on GCE node
I20260924 16:10:57.094094 22799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:10:57.095515 22799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:10:57.096930 22799 hybrid_clock.cc:648] HybridClock initialized: now 1790266257096822 us; error 89 us; skew 500 ppm
I20260924 16:10:57.097448 22799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:10:57 dist-test-slave-1gc3 krb5kdc[22423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266257, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:10:57.101816 22799 init.cc:377] Logged in from keytab as kudu/127.3.214.195@KRBTEST.COM (short username kudu)
I20260924 16:10:57.104004 22799 webserver.cc:460] Webserver started at http://127.3.214.195:44223/ using document root <none> and password file <none>
I20260924 16:10:57.104959 22799 fs_manager.cc:359] Metadata directory not provided
I20260924 16:10:57.105104 22799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:10:57.105520 22799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:10:57.109344 22799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "f44ebc3acacb4dddacfbdfb203bd1af6"
format_stamp: "Formatted at 2026-09-24 16:10:57 on dist-test-slave-1gc3"
server_key: "64ee8250706431b49f300948815099b2"
server_key_iv: "c0a4d7be27173ac540c1dd416ee32fc3"
server_key_version: "encryptionkey@0"
I20260924 16:10:57.110370 22799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "f44ebc3acacb4dddacfbdfb203bd1af6"
format_stamp: "Formatted at 2026-09-24 16:10:57 on dist-test-slave-1gc3"
server_key: "64ee8250706431b49f300948815099b2"
server_key_iv: "c0a4d7be27173ac540c1dd416ee32fc3"
server_key_version: "encryptionkey@0"
I20260924 16:10:57.117162 22799 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260924 16:10:57.121714 22814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:57.123680 22799 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:10:57.123910 22799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "f44ebc3acacb4dddacfbdfb203bd1af6"
format_stamp: "Formatted at 2026-09-24 16:10:57 on dist-test-slave-1gc3"
server_key: "64ee8250706431b49f300948815099b2"
server_key_iv: "c0a4d7be27173ac540c1dd416ee32fc3"
server_key_version: "encryptionkey@0"
I20260924 16:10:57.124171 22799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:10:57.160724 22799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:10:57.166467 22799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:10:57.166797 22799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:10:57.168083 22799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:10:57.169778 22799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:10:57.169867 22799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:57.170012 22799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:10:57.170069 22799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:57.209619 22799 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:37053
I20260924 16:10:57.209682 22927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:37053 every 8 connection(s)
I20260924 16:10:57.211441 22799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:10:57.215912 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 22799
I20260924 16:10:57.216143 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:10:57.216759 3931 external_mini_cluster.cc:1428] Setting key 4ec4a87a5a4e1b9eb51a2362ab7ab398
Sep 24 16:10:57 dist-test-slave-1gc3 krb5kdc[22423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266257, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:10:57.237277 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:57.214897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:53717 (local address 127.3.214.254:45515)
0924 16:10:57.215404 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:57.215408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:57.216991 (+ 1583us) server_negotiation.cc:407] Connection header received
0924 16:10:57.219518 (+ 2527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:57.219523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:57.219637 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:57.219751 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:57.221769 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.222744 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.224599 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.225103 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.225237 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:57.229232 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:57.229252 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:57.229259 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:57.229328 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:57.233520 (+ 4192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:57.234654 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:57.234659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:57.234662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:57.234819 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:57.235299 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:57.235303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:57.235305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:57.235622 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:57.235781 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:57.236589 (+ 808us) server_negotiation.cc:299] Negotiation successful
0924 16:10:57.236866 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":276,"threads_started":1}
I20260924 16:10:57.238442 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:57.215530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:45515 (local address 127.3.214.195:53717)
0924 16:10:57.216685 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:57.216794 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:57.219167 (+ 2373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:57.220113 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:57.220126 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:57.220845 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:10:57.221541 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:57.221563 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.222924 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.222929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:57.224385 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:57.224404 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.225361 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.225366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:57.225613 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:57.226411 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:10:57.226443 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:10:57.228884 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:10:57.233772 (+ 4888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:10:57.233786 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:10:57.233810 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:10:57.234499 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:10:57.234946 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:10:57.234952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:10:57.234956 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:10:57.235173 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:10:57.235764 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:10:57.235773 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:10:57.236175 (+ 402us) client_negotiation.cc:769] Sending connection context
0924 16:10:57.236436 (+ 261us) client_negotiation.cc:240] Negotiation successful
0924 16:10:57.237051 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":432,"threads_started":1}
I20260924 16:10:57.240500 22928 heartbeater.cc:347] Connected to a master server at 127.3.214.254:45515
I20260924 16:10:57.240906 22928 heartbeater.cc:464] Registering TS with master...
I20260924 16:10:57.241765 22928 heartbeater.cc:511] Master 127.3.214.254:45515 requested a full tablet report, sending...
I20260924 16:10:57.243628 22468 ts_manager.cc:194] Registered new tserver with Master: f44ebc3acacb4dddacfbdfb203bd1af6 (127.3.214.195:37053)
I20260924 16:10:57.244760 22468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:53717
I20260924 16:10:57.247622 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:10:57.268788 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:57.252707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37746 (local address 127.3.214.254:45515)
0924 16:10:57.252982 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:57.252989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:57.253392 (+ 403us) server_negotiation.cc:407] Connection header received
0924 16:10:57.253572 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:57.253578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:57.253707 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:57.253899 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:57.255430 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.256606 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.258181 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.258875 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.259030 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:57.262309 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:57.262315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:57.262319 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:57.262362 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:57.265306 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:57.266477 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:57.266483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:57.266486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:57.266657 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:57.267134 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:57.267138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:57.267140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:57.267555 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:57.267818 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:57.268296 (+ 478us) server_negotiation.cc:299] Negotiation successful
0924 16:10:57.268490 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260924 16:10:57.274587 22468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:10:57.282859 22468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:10:57.313498 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:57.300709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46966 (local address 127.3.214.194:45523)
0924 16:10:57.302187 (+ 1478us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:57.302199 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:57.302239 (+ 40us) server_negotiation.cc:407] Connection header received
0924 16:10:57.302368 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:57.302377 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:57.302717 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:57.303814 (+ 1097us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:10:57.304150 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.305959 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.310048 (+ 4089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.311941 (+ 1893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.312124 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:57.312374 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:57.312884 (+ 510us) server_negotiation.cc:299] Negotiation successful
0924 16:10:57.313108 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1226,"thread_start_us":1220,"threads_started":1}
I20260924 16:10:57.313841 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:57.299919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:44591 (local address 127.0.0.1:58292)
0924 16:10:57.301002 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:57.302910 (+ 1908us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:57.303037 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:57.304042 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:57.304047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:57.304074 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:10:57.304517 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:57.304531 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.306696 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.306701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:57.310019 (+ 3318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:57.310033 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.312606 (+ 2573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.312613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:57.312784 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:57.312827 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 16:10:57.312937 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 16:10:57.313165 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":730,"thread_start_us":626,"threads_started":1}
I20260924 16:10:57.314304 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:57.300148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:45523 (local address 127.0.0.1:46966)
0924 16:10:57.301002 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:57.301081 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:57.301330 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:57.303029 (+ 1699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:57.303037 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:57.303066 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:10:57.303618 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:57.303634 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.306152 (+ 2518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.306158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:57.309318 (+ 3160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:57.309346 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.312221 (+ 2875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.312229 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:57.312547 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:57.312622 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 16:10:57.312787 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 16:10:57.313447 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":382,"spinlock_wait_cycles":1280,"thread_start_us":775,"threads_started":1}
I20260924 16:10:57.315078 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:57.300385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58292 (local address 127.3.214.193:44591)
0924 16:10:57.302583 (+ 2198us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:57.302591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:57.303155 (+ 564us) server_negotiation.cc:407] Connection header received
0924 16:10:57.303268 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:57.303275 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:57.303551 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:57.303758 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:10:57.304769 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.306459 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.310339 (+ 3880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.311941 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.312124 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:57.312497 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:57.313150 (+ 653us) server_negotiation.cc:299] Negotiation successful
0924 16:10:57.314713 (+ 1563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2017,"thread_start_us":2232,"threads_started":1}
I20260924 16:10:57.319304 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:57.302429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:37053 (local address 127.0.0.1:33806)
0924 16:10:57.309353 (+ 6924us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:57.309379 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:57.309689 (+ 310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:57.310916 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:57.310924 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:57.310956 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:10:57.311360 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:57.311373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.313309 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.313314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:57.315966 (+ 2652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:57.316158 (+ 192us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.318321 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.318330 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:57.318533 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:57.318580 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 16:10:57.318726 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 16:10:57.318897 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6748,"thread_start_us":6531,"threads_started":1}
I20260924 16:10:57.319396 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:57.302834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33806 (local address 127.3.214.195:37053)
0924 16:10:57.309953 (+ 7119us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:57.309969 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:57.310019 (+ 50us) server_negotiation.cc:407] Connection header received
0924 16:10:57.310254 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:57.310264 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:57.310651 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:57.310916 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:10:57.311564 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.313107 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.316407 (+ 3300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.317981 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.318174 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:57.318455 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:57.318810 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 16:10:57.319008 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6806,"thread_start_us":7219,"threads_started":1}
I20260924 16:10:57.319609 22728 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce4a067ef8846e6a6f6f38f332e1c2d (DEFAULT_TABLE table=test-table [id=b5a2844787744334b6033c8a55b759c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:10:57.320118 22592 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce4a067ef8846e6a6f6f38f332e1c2d (DEFAULT_TABLE table=test-table [id=b5a2844787744334b6033c8a55b759c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:10:57.322000 22728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ce4a067ef8846e6a6f6f38f332e1c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:10:57.322162 22592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ce4a067ef8846e6a6f6f38f332e1c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:10:57.324810 22862 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce4a067ef8846e6a6f6f38f332e1c2d (DEFAULT_TABLE table=test-table [id=b5a2844787744334b6033c8a55b759c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:10:57.326767 22862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ce4a067ef8846e6a6f6f38f332e1c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:10:57.337255 22947 tablet_bootstrap.cc:492] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6: Bootstrap starting.
I20260924 16:10:57.341638 22948 tablet_bootstrap.cc:492] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 3323ef1ac8254f42a936ec0c7239fe09: Bootstrap starting.
I20260924 16:10:57.343042 22947 tablet_bootstrap.cc:654] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6: Neither blocks nor log segments found. Creating new log.
I20260924 16:10:57.343712 22949 tablet_bootstrap.cc:492] T 5ce4a067ef8846e6a6f6f38f332e1c2d P f44ebc3acacb4dddacfbdfb203bd1af6: Bootstrap starting.
I20260924 16:10:57.344750 22947 log.cc:824] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6: Log is configured to *not* fsync() on all Append() calls
I20260924 16:10:57.346170 22948 tablet_bootstrap.cc:654] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 3323ef1ac8254f42a936ec0c7239fe09: Neither blocks nor log segments found. Creating new log.
I20260924 16:10:57.347577 22948 log.cc:824] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 3323ef1ac8254f42a936ec0c7239fe09: Log is configured to *not* fsync() on all Append() calls
I20260924 16:10:57.348086 22949 tablet_bootstrap.cc:654] T 5ce4a067ef8846e6a6f6f38f332e1c2d P f44ebc3acacb4dddacfbdfb203bd1af6: Neither blocks nor log segments found. Creating new log.
I20260924 16:10:57.349727 22949 log.cc:824] T 5ce4a067ef8846e6a6f6f38f332e1c2d P f44ebc3acacb4dddacfbdfb203bd1af6: Log is configured to *not* fsync() on all Append() calls
I20260924 16:10:57.349888 22947 tablet_bootstrap.cc:492] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6: No bootstrap required, opened a new log
I20260924 16:10:57.350381 22947 ts_tablet_manager.cc:1397] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260924 16:10:57.352629 22948 tablet_bootstrap.cc:492] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 3323ef1ac8254f42a936ec0c7239fe09: No bootstrap required, opened a new log
I20260924 16:10:57.352993 22948 ts_tablet_manager.cc:1397] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 3323ef1ac8254f42a936ec0c7239fe09: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20260924 16:10:57.357175 22949 tablet_bootstrap.cc:492] T 5ce4a067ef8846e6a6f6f38f332e1c2d P f44ebc3acacb4dddacfbdfb203bd1af6: No bootstrap required, opened a new log
I20260924 16:10:57.357506 22949 ts_tablet_manager.cc:1397] T 5ce4a067ef8846e6a6f6f38f332e1c2d P f44ebc3acacb4dddacfbdfb203bd1af6: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260924 16:10:57.362722 22947 raft_consensus.cc:348] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323ef1ac8254f42a936ec0c7239fe09" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 44591 } } peers { permanent_uuid: "2817261e09c94665a4a44684864d88c6" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45523 } } peers { permanent_uuid: "f44ebc3acacb4dddacfbdfb203bd1af6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37053 } }
I20260924 16:10:57.363435 22947 raft_consensus.cc:374] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:10:57.363557 22947 raft_consensus.cc:729] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2817261e09c94665a4a44684864d88c6, State: Initialized, Role: FOLLOWER
I20260924 16:10:57.364481 22947 consensus_queue.cc:260] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323ef1ac8254f42a936ec0c7239fe09" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 44591 } } peers { permanent_uuid: "2817261e09c94665a4a44684864d88c6" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45523 } } peers { permanent_uuid: "f44ebc3acacb4dddacfbdfb203bd1af6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37053 } }
I20260924 16:10:57.364768 22948 raft_consensus.cc:348] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 3323ef1ac8254f42a936ec0c7239fe09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323ef1ac8254f42a936ec0c7239fe09" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 44591 } } peers { permanent_uuid: "2817261e09c94665a4a44684864d88c6" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45523 } } peers { permanent_uuid: "f44ebc3acacb4dddacfbdfb203bd1af6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37053 } }
I20260924 16:10:57.365375 22948 raft_consensus.cc:374] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 3323ef1ac8254f42a936ec0c7239fe09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:10:57.365487 22948 raft_consensus.cc:729] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 3323ef1ac8254f42a936ec0c7239fe09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3323ef1ac8254f42a936ec0c7239fe09, State: Initialized, Role: FOLLOWER
I20260924 16:10:57.366261 22793 heartbeater.cc:503] Master 127.3.214.254:45515 was elected leader, sending a full tablet report...
I20260924 16:10:57.366276 22949 raft_consensus.cc:348] T 5ce4a067ef8846e6a6f6f38f332e1c2d P f44ebc3acacb4dddacfbdfb203bd1af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323ef1ac8254f42a936ec0c7239fe09" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 44591 } } peers { permanent_uuid: "2817261e09c94665a4a44684864d88c6" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45523 } } peers { permanent_uuid: "f44ebc3acacb4dddacfbdfb203bd1af6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37053 } }
I20260924 16:10:57.366449 22948 consensus_queue.cc:260] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 3323ef1ac8254f42a936ec0c7239fe09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323ef1ac8254f42a936ec0c7239fe09" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 44591 } } peers { permanent_uuid: "2817261e09c94665a4a44684864d88c6" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45523 } } peers { permanent_uuid: "f44ebc3acacb4dddacfbdfb203bd1af6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37053 } }
I20260924 16:10:57.366806 22949 raft_consensus.cc:374] T 5ce4a067ef8846e6a6f6f38f332e1c2d P f44ebc3acacb4dddacfbdfb203bd1af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:10:57.366905 22949 raft_consensus.cc:729] T 5ce4a067ef8846e6a6f6f38f332e1c2d P f44ebc3acacb4dddacfbdfb203bd1af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f44ebc3acacb4dddacfbdfb203bd1af6, State: Initialized, Role: FOLLOWER
I20260924 16:10:57.367138 22947 ts_tablet_manager.cc:1428] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
I20260924 16:10:57.367749 22949 consensus_queue.cc:260] T 5ce4a067ef8846e6a6f6f38f332e1c2d P f44ebc3acacb4dddacfbdfb203bd1af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323ef1ac8254f42a936ec0c7239fe09" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 44591 } } peers { permanent_uuid: "2817261e09c94665a4a44684864d88c6" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45523 } } peers { permanent_uuid: "f44ebc3acacb4dddacfbdfb203bd1af6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37053 } }
I20260924 16:10:57.369184 22948 ts_tablet_manager.cc:1428] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 3323ef1ac8254f42a936ec0c7239fe09: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260924 16:10:57.370534 22928 heartbeater.cc:503] Master 127.3.214.254:45515 was elected leader, sending a full tablet report...
I20260924 16:10:57.371874 22949 ts_tablet_manager.cc:1428] T 5ce4a067ef8846e6a6f6f38f332e1c2d P f44ebc3acacb4dddacfbdfb203bd1af6: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
W20260924 16:10:57.463758 22929 tablet.cc:2369] T 5ce4a067ef8846e6a6f6f38f332e1c2d P f44ebc3acacb4dddacfbdfb203bd1af6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:10:57.482842 22659 tablet.cc:2369] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 3323ef1ac8254f42a936ec0c7239fe09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:10:57.499536 22953 raft_consensus.cc:482] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:10:57.499965 22953 raft_consensus.cc:504] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323ef1ac8254f42a936ec0c7239fe09" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 44591 } } peers { permanent_uuid: "2817261e09c94665a4a44684864d88c6" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45523 } } peers { permanent_uuid: "f44ebc3acacb4dddacfbdfb203bd1af6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37053 } }
I20260924 16:10:57.502279 22953 leader_election.cc:288] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3323ef1ac8254f42a936ec0c7239fe09 (127.3.214.193:44591), f44ebc3acacb4dddacfbdfb203bd1af6 (127.3.214.195:37053)
I20260924 16:10:57.511106 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:57.502958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:49533 (local address 127.3.214.195:37053)
0924 16:10:57.503274 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:57.503283 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:57.503920 (+ 637us) server_negotiation.cc:407] Connection header received
0924 16:10:57.504223 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:57.504230 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:57.504349 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:57.504442 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:10:57.505221 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.506354 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.509058 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.509932 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.510127 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:57.510223 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:57.510666 (+ 443us) server_negotiation.cc:299] Negotiation successful
0924 16:10:57.510828 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 16:10:57.511184 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:57.502974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:51511 (local address 127.3.214.193:44591)
0924 16:10:57.503526 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:57.503533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:57.504326 (+ 793us) server_negotiation.cc:407] Connection header received
0924 16:10:57.504589 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:57.504595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:57.504690 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:57.504810 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:10:57.505734 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.506638 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.509076 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.509913 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.510091 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:57.510203 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:57.510726 (+ 523us) server_negotiation.cc:299] Negotiation successful
0924 16:10:57.510885 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346}
I20260924 16:10:57.511408 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:57.502787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:37053 (local address 127.3.214.194:49533)
0924 16:10:57.503746 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:57.503773 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:57.504031 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:57.504514 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:57.504519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:57.504556 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:10:57.505045 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:57.505061 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.506525 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.506529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:57.508890 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:57.508905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.510280 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.510288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:57.510456 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:57.510527 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 16:10:57.510616 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 16:10:57.510798 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":177,"thread_start_us":460,"threads_started":1}
I20260924 16:10:57.511718 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:57.502974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:44591 (local address 127.3.214.194:51511)
0924 16:10:57.504119 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:57.504141 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:57.504353 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:57.504882 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:57.504887 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:57.504916 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:10:57.505495 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:57.505512 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.506835 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.506841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:57.508874 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:57.508893 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.510326 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.510334 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:57.510547 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:57.510581 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 16:10:57.510674 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 16:10:57.510850 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"mutex_wait_us":39,"thread_start_us":703,"threads_started":1}
I20260924 16:10:57.512727 22882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce4a067ef8846e6a6f6f38f332e1c2d" candidate_uuid: "2817261e09c94665a4a44684864d88c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f44ebc3acacb4dddacfbdfb203bd1af6" is_pre_election: true
I20260924 16:10:57.513000 22612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce4a067ef8846e6a6f6f38f332e1c2d" candidate_uuid: "2817261e09c94665a4a44684864d88c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3323ef1ac8254f42a936ec0c7239fe09" is_pre_election: true
I20260924 16:10:57.513286 22882 raft_consensus.cc:2454] T 5ce4a067ef8846e6a6f6f38f332e1c2d P f44ebc3acacb4dddacfbdfb203bd1af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2817261e09c94665a4a44684864d88c6 in term 0.
I20260924 16:10:57.513657 22612 raft_consensus.cc:2454] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 3323ef1ac8254f42a936ec0c7239fe09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2817261e09c94665a4a44684864d88c6 in term 0.
I20260924 16:10:57.514074 22681 leader_election.cc:302] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2817261e09c94665a4a44684864d88c6, f44ebc3acacb4dddacfbdfb203bd1af6; no voters:
I20260924 16:10:57.514607 22953 raft_consensus.cc:2781] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:10:57.514760 22953 raft_consensus.cc:482] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:10:57.514871 22953 raft_consensus.cc:3037] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:10:57.517412 22953 raft_consensus.cc:504] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323ef1ac8254f42a936ec0c7239fe09" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 44591 } } peers { permanent_uuid: "2817261e09c94665a4a44684864d88c6" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45523 } } peers { permanent_uuid: "f44ebc3acacb4dddacfbdfb203bd1af6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37053 } }
I20260924 16:10:57.518237 22953 leader_election.cc:288] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6 [CANDIDATE]: Term 1 election: Requested vote from peers 3323ef1ac8254f42a936ec0c7239fe09 (127.3.214.193:44591), f44ebc3acacb4dddacfbdfb203bd1af6 (127.3.214.195:37053)
I20260924 16:10:57.518914 22612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce4a067ef8846e6a6f6f38f332e1c2d" candidate_uuid: "2817261e09c94665a4a44684864d88c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3323ef1ac8254f42a936ec0c7239fe09"
I20260924 16:10:57.519040 22882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce4a067ef8846e6a6f6f38f332e1c2d" candidate_uuid: "2817261e09c94665a4a44684864d88c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f44ebc3acacb4dddacfbdfb203bd1af6"
I20260924 16:10:57.519254 22612 raft_consensus.cc:3037] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 3323ef1ac8254f42a936ec0c7239fe09 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:10:57.519254 22882 raft_consensus.cc:3037] T 5ce4a067ef8846e6a6f6f38f332e1c2d P f44ebc3acacb4dddacfbdfb203bd1af6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:10:57.522732 22612 raft_consensus.cc:2454] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 3323ef1ac8254f42a936ec0c7239fe09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2817261e09c94665a4a44684864d88c6 in term 1.
I20260924 16:10:57.522732 22882 raft_consensus.cc:2454] T 5ce4a067ef8846e6a6f6f38f332e1c2d P f44ebc3acacb4dddacfbdfb203bd1af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2817261e09c94665a4a44684864d88c6 in term 1.
I20260924 16:10:57.523406 22681 leader_election.cc:302] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2817261e09c94665a4a44684864d88c6, 3323ef1ac8254f42a936ec0c7239fe09, f44ebc3acacb4dddacfbdfb203bd1af6; no voters:
I20260924 16:10:57.523856 22953 raft_consensus.cc:2781] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:10:57.524457 22953 raft_consensus.cc:686] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6 [term 1 LEADER]: Becoming Leader. State: Replica: 2817261e09c94665a4a44684864d88c6, State: Running, Role: LEADER
I20260924 16:10:57.525041 22953 consensus_queue.cc:237] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323ef1ac8254f42a936ec0c7239fe09" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 44591 } } peers { permanent_uuid: "2817261e09c94665a4a44684864d88c6" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45523 } } peers { permanent_uuid: "f44ebc3acacb4dddacfbdfb203bd1af6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37053 } }
I20260924 16:10:57.530422 22468 catalog_manager.cc:5495] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2817261e09c94665a4a44684864d88c6 (127.3.214.194). New cstate: current_term: 1 leader_uuid: "2817261e09c94665a4a44684864d88c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323ef1ac8254f42a936ec0c7239fe09" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 44591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2817261e09c94665a4a44684864d88c6" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f44ebc3acacb4dddacfbdfb203bd1af6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37053 } health_report { overall_health: UNKNOWN } } }
W20260924 16:10:57.560065 22794 tablet.cc:2369] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:10:57.571228 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:57.563599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46972 (local address 127.3.214.194:45523)
0924 16:10:57.563824 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:57.563829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:57.563860 (+ 31us) server_negotiation.cc:407] Connection header received
0924 16:10:57.564152 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:57.564158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:57.564299 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:57.564479 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:10:57.565009 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.565943 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.568170 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.568702 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.568822 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:57.569616 (+ 794us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:10:57.570393 (+ 777us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:10:57.570563 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:57.570771 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 16:10:57.570832 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:10:57.581634 22882 raft_consensus.cc:1261] T 5ce4a067ef8846e6a6f6f38f332e1c2d P f44ebc3acacb4dddacfbdfb203bd1af6 [term 1 FOLLOWER]: Refusing update from remote peer 2817261e09c94665a4a44684864d88c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:10:57.581634 22612 raft_consensus.cc:1261] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 3323ef1ac8254f42a936ec0c7239fe09 [term 1 FOLLOWER]: Refusing update from remote peer 2817261e09c94665a4a44684864d88c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:10:57.582983 22958 consensus_queue.cc:1035] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f44ebc3acacb4dddacfbdfb203bd1af6" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 37053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:10:57.583529 22953 consensus_queue.cc:1035] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3323ef1ac8254f42a936ec0c7239fe09" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 44591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:10:57.616308 22468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:10:57.616658 22468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:10:57.621493 22468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63243771ca1f43c4ade500a8732c0d8b: Sending DeleteTablet for 3 replicas of tablet 5ce4a067ef8846e6a6f6f38f332e1c2d
I20260924 16:10:57.623266 22592 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ce4a067ef8846e6a6f6f38f332e1c2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:10:57 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:58292
I20260924 16:10:57.623392 22728 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ce4a067ef8846e6a6f6f38f332e1c2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:10:57 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:46966
I20260924 16:10:57.624439 22971 tablet_replica.cc:330] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 3323ef1ac8254f42a936ec0c7239fe09: stopping tablet replica
I20260924 16:10:57.624828 22862 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ce4a067ef8846e6a6f6f38f332e1c2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:10:57 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:33806
I20260924 16:10:57.625296 22971 raft_consensus.cc:2229] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 3323ef1ac8254f42a936ec0c7239fe09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:10:57.625519 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 22529
I20260924 16:10:57.626214 22971 raft_consensus.cc:2258] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 3323ef1ac8254f42a936ec0c7239fe09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:10:57.630294 22971 ts_tablet_manager.cc:1905] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 3323ef1ac8254f42a936ec0c7239fe09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:10:57.635752 22973 tablet_replica.cc:330] T 5ce4a067ef8846e6a6f6f38f332e1c2d P f44ebc3acacb4dddacfbdfb203bd1af6: stopping tablet replica
I20260924 16:10:57.636418 22973 raft_consensus.cc:2229] T 5ce4a067ef8846e6a6f6f38f332e1c2d P f44ebc3acacb4dddacfbdfb203bd1af6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:10:57.637086 22973 raft_consensus.cc:2258] T 5ce4a067ef8846e6a6f6f38f332e1c2d P f44ebc3acacb4dddacfbdfb203bd1af6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:10:57.638289 22972 tablet_replica.cc:330] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6: stopping tablet replica
I20260924 16:10:57.638834 22972 raft_consensus.cc:2229] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:10:57.639505 22971 ts_tablet_manager.cc:1918] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 3323ef1ac8254f42a936ec0c7239fe09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:10:57.639703 22972 raft_consensus.cc:2258] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:10:57.639843 22971 log.cc:1196] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 3323ef1ac8254f42a936ec0c7239fe09: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-0/wal/wals/5ce4a067ef8846e6a6f6f38f332e1c2d
I20260924 16:10:57.640547 22971 ts_tablet_manager.cc:1939] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 3323ef1ac8254f42a936ec0c7239fe09: Deleting consensus metadata
I20260924 16:10:57.642424 22973 ts_tablet_manager.cc:1905] T 5ce4a067ef8846e6a6f6f38f332e1c2d P f44ebc3acacb4dddacfbdfb203bd1af6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:10:57.643163 22455 catalog_manager.cc:4841] TS 3323ef1ac8254f42a936ec0c7239fe09 (127.3.214.193:44591): tablet 5ce4a067ef8846e6a6f6f38f332e1c2d (table test-table [id=b5a2844787744334b6033c8a55b759c3]) successfully deleted
I20260924 16:10:57.647737 22972 ts_tablet_manager.cc:1905] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:10:57.650440 22973 ts_tablet_manager.cc:1918] T 5ce4a067ef8846e6a6f6f38f332e1c2d P f44ebc3acacb4dddacfbdfb203bd1af6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:10:57.650707 22973 log.cc:1196] T 5ce4a067ef8846e6a6f6f38f332e1c2d P f44ebc3acacb4dddacfbdfb203bd1af6: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/5ce4a067ef8846e6a6f6f38f332e1c2d
I20260924 16:10:57.651675 22973 ts_tablet_manager.cc:1939] T 5ce4a067ef8846e6a6f6f38f332e1c2d P f44ebc3acacb4dddacfbdfb203bd1af6: Deleting consensus metadata
I20260924 16:10:57.654366 22453 catalog_manager.cc:4841] TS f44ebc3acacb4dddacfbdfb203bd1af6 (127.3.214.195:37053): tablet 5ce4a067ef8846e6a6f6f38f332e1c2d (table test-table [id=b5a2844787744334b6033c8a55b759c3]) successfully deleted
Sep 24 16:10:57 dist-test-slave-1gc3 krb5kdc[22423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.193@KRBTEST.COM
I20260924 16:10:57.657115 22972 ts_tablet_manager.cc:1918] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:10:57.657356 22972 log.cc:1196] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/5ce4a067ef8846e6a6f6f38f332e1c2d
I20260924 16:10:57.657965 22972 ts_tablet_manager.cc:1939] T 5ce4a067ef8846e6a6f6f38f332e1c2d P 2817261e09c94665a4a44684864d88c6: Deleting consensus metadata
I20260924 16:10:57.660254 22452 catalog_manager.cc:4841] TS 2817261e09c94665a4a44684864d88c6 (127.3.214.194:45523): tablet 5ce4a067ef8846e6a6f6f38f332e1c2d (table test-table [id=b5a2844787744334b6033c8a55b759c3]) successfully deleted
I20260924 16:10:57.662882 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:57.629219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58294 (local address 127.3.214.193:44591)
0924 16:10:57.629434 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:57.629441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:57.642459 (+ 13018us) server_negotiation.cc:407] Connection header received
0924 16:10:57.643022 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:57.643029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:57.643125 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:57.643220 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:57.645280 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.646501 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.648884 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.649572 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.649688 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:57.656085 (+ 6397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:57.656108 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:57.656131 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:57.656182 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:57.659475 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:57.660446 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:57.660458 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:57.660461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:57.660633 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:57.661025 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:57.661030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:57.661033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:57.661509 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:57.661843 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:57.662206 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 16:10:57.662489 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 16:10:57.663861 22653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:10:57.759142 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 22529
I20260924 16:10:57.783776 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 22664
Sep 24 16:10:57 dist-test-slave-1gc3 krb5kdc[22423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.194@KRBTEST.COM
I20260924 16:10:57.803329 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:57.784687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.3.214.194:45523)
0924 16:10:57.784904 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:57.784911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:57.785041 (+ 130us) server_negotiation.cc:407] Connection header received
0924 16:10:57.785274 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:57.785280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:57.785352 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:57.785445 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:57.786864 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.787770 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.789074 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.789552 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.789672 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:57.794711 (+ 5039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:57.794738 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:57.794776 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:57.794845 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:57.799459 (+ 4614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:57.800511 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:57.800528 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:57.800532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:57.800758 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:57.801245 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:57.801251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:57.801255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:57.801852 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:57.802272 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:57.802567 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 16:10:57.802885 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 16:10:57.804394 22788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:10:57.927608 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 22664
I20260924 16:10:57.949340 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 22799
Sep 24 16:10:57 dist-test-slave-1gc3 krb5kdc[22423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.195@KRBTEST.COM
I20260924 16:10:57.966948 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:57.950258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33816 (local address 127.3.214.195:37053)
0924 16:10:57.950499 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:57.950505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:57.950528 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:10:57.950781 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:57.950787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:57.950887 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:57.950994 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:57.952534 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.953422 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.954972 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:57.955427 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:57.955551 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:57.960691 (+ 5140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:57.960715 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:57.960736 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:10:57.960792 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:57.963703 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:57.964717 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:57.964726 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:57.964728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:57.964898 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:10:57.965345 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:10:57.965349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:10:57.965351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:10:57.965760 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:57.966013 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:57.966397 (+ 384us) server_negotiation.cc:299] Negotiation successful
0924 16:10:57.966616 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 16:10:57.967854 22922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:10:58.069379 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 22799
I20260924 16:10:58.087368 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 22436
I20260924 16:10:58.088771 22498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:10:58.211233 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 22436
2026-09-24T16:10:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4902 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T16:10:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:10:58Z Disabled control of system clock
I20260924 16:10:58.301781 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:36393
--webserver_interface=127.3.214.254
--webserver_port=0
--builtin_ntp_servers=127.3.214.212:37471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:36393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:10:58.698078 22982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:10:58.698519 22982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:10:58.698619 22982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:10:58.707669 22982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:10:58.707860 22982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:10:58.707903 22982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:10:58.707960 22982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:10:58.708011 22982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:10:58.719955 22982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:37471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:36393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:36393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:10:58.721849 22982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:10:58.724449 22982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:10:58.739123 22987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:58.739203 22988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:58.739096 22990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:58.740139 22982 server_base.cc:1034] running on GCE node
I20260924 16:10:58.741161 22982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:10:58.742954 22982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:10:58.744246 22982 hybrid_clock.cc:648] HybridClock initialized: now 1790266258744148 us; error 81 us; skew 500 ppm
I20260924 16:10:58.744704 22982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:10:58.747601 22982 webserver.cc:460] Webserver started at http://127.3.214.254:44685/ using document root <none> and password file <none>
I20260924 16:10:58.748448 22982 fs_manager.cc:359] Metadata directory not provided
I20260924 16:10:58.748553 22982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:10:58.748906 22982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:10:58.752380 22982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "15d7ab100e624b68a3cc492edd012e0e"
format_stamp: "Formatted at 2026-09-24 16:10:58 on dist-test-slave-1gc3"
server_key: "d8848ac5f44035ae03fd809b7131ed7d"
server_key_iv: "153e91dc69722794c8caa59b17e014bf"
server_key_version: "encryptionkey@0"
I20260924 16:10:58.753463 22982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "15d7ab100e624b68a3cc492edd012e0e"
format_stamp: "Formatted at 2026-09-24 16:10:58 on dist-test-slave-1gc3"
server_key: "d8848ac5f44035ae03fd809b7131ed7d"
server_key_iv: "153e91dc69722794c8caa59b17e014bf"
server_key_version: "encryptionkey@0"
I20260924 16:10:58.760293 22982 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.003s
I20260924 16:10:58.765317 22996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:58.767391 22982 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260924 16:10:58.767670 22982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "15d7ab100e624b68a3cc492edd012e0e"
format_stamp: "Formatted at 2026-09-24 16:10:58 on dist-test-slave-1gc3"
server_key: "d8848ac5f44035ae03fd809b7131ed7d"
server_key_iv: "153e91dc69722794c8caa59b17e014bf"
server_key_version: "encryptionkey@0"
I20260924 16:10:58.767932 22982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:10:58.796308 22982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:10:58.797406 22982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:10:58.797727 22982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:10:58.820408 23048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:36393 every 8 connection(s)
I20260924 16:10:58.820475 22982 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:36393
I20260924 16:10:58.822188 22982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:10:58.824000 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 22982
I20260924 16:10:58.824254 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:10:58.824994 3931 external_mini_cluster.cc:1428] Setting key f2aea0efde6a1f8429d7aab15b1bc757
I20260924 16:10:58.828860 23049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:10:58.841856 23052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:58.827732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41688 (local address 127.3.214.254:36393)
0924 16:10:58.829260 (+ 1528us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:58.829275 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:58.829335 (+ 60us) server_negotiation.cc:407] Connection header received
0924 16:10:58.831268 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:58.831299 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:58.831972 (+ 673us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:58.832472 (+ 500us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:58.833143 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:58.834700 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:58.836302 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:58.837084 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:58.837281 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:58.838880 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:58.838889 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:58.838910 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:10:58.839643 (+ 733us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:58.839745 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:10:58.839773 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:58.839925 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:58.840265 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 16:10:58.840697 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":505,"threads_started":1}
I20260924 16:10:58.844127 23049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e: Bootstrap starting.
I20260924 16:10:58.848579 23049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e: Neither blocks nor log segments found. Creating new log.
I20260924 16:10:58.849994 23049 log.cc:824] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:10:58.853915 23049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e: No bootstrap required, opened a new log
I20260924 16:10:58.863284 23049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d7ab100e624b68a3cc492edd012e0e" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 36393 } }
I20260924 16:10:58.863945 23049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:10:58.864073 23049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15d7ab100e624b68a3cc492edd012e0e, State: Initialized, Role: FOLLOWER
I20260924 16:10:58.864890 23049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d7ab100e624b68a3cc492edd012e0e" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 36393 } }
I20260924 16:10:58.865170 23049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:10:58.865343 23049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:10:58.865548 23049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:10:58.868357 23049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d7ab100e624b68a3cc492edd012e0e" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 36393 } }
I20260924 16:10:58.869097 23049 leader_election.cc:302] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15d7ab100e624b68a3cc492edd012e0e; no voters:
I20260924 16:10:58.869977 23049 leader_election.cc:288] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:10:58.870296 23054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:10:58.871444 23054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e [term 1 LEADER]: Becoming Leader. State: Replica: 15d7ab100e624b68a3cc492edd012e0e, State: Running, Role: LEADER
I20260924 16:10:58.872215 23054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d7ab100e624b68a3cc492edd012e0e" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 36393 } }
I20260924 16:10:58.873317 23049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:10:58.876251 23056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15d7ab100e624b68a3cc492edd012e0e. Latest consensus state: current_term: 1 leader_uuid: "15d7ab100e624b68a3cc492edd012e0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d7ab100e624b68a3cc492edd012e0e" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 36393 } } }
I20260924 16:10:58.876892 23056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e [sys.catalog]: This master's current role is: LEADER
I20260924 16:10:58.876317 23055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15d7ab100e624b68a3cc492edd012e0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d7ab100e624b68a3cc492edd012e0e" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 36393 } } }
I20260924 16:10:58.877190 23055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e [sys.catalog]: This master's current role is: LEADER
I20260924 16:10:58.881223 23060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:10:58.888711 23060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:10:58.902181 23060 catalog_manager.cc:1347] Generated new cluster ID: f978a38b090d4c54a0c3b4b965d01ea3
I20260924 16:10:58.902351 23060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:10:58.928144 23060 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:10:58.930655 23060 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:10:58.944476 23060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e: Generated new TSK 0
I20260924 16:10:58.945547 23060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:10:58.962700 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:36393
--builtin_ntp_servers=127.3.214.212:37471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:10:59.398697 23073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:10:59.399017 23073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:10:59.399130 23073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:10:59.407915 23073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:10:59.408064 23073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:10:59.408200 23073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:10:59.418049 23073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:37471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:36393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:10:59.419525 23073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:10:59.421564 23073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:10:59.436985 23079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:59.437187 23081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:59.437242 23078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:59.439062 23073 server_base.cc:1034] running on GCE node
I20260924 16:10:59.439939 23073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:10:59.441139 23073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:10:59.442411 23073 hybrid_clock.cc:648] HybridClock initialized: now 1790266259442330 us; error 66 us; skew 500 ppm
I20260924 16:10:59.442827 23073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:10:59.445647 23073 webserver.cc:460] Webserver started at http://127.3.214.193:33039/ using document root <none> and password file <none>
I20260924 16:10:59.446524 23073 fs_manager.cc:359] Metadata directory not provided
I20260924 16:10:59.446643 23073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:10:59.447012 23073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:10:59.450353 23073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "acfd131ec733487d9ab8894f2ed5f62a"
format_stamp: "Formatted at 2026-09-24 16:10:59 on dist-test-slave-1gc3"
server_key: "068705f79343a0aa5bb6bf76c6a37057"
server_key_iv: "f1fccb54cdc37dc1fbde91caad77324d"
server_key_version: "encryptionkey@0"
I20260924 16:10:59.451252 23073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "acfd131ec733487d9ab8894f2ed5f62a"
format_stamp: "Formatted at 2026-09-24 16:10:59 on dist-test-slave-1gc3"
server_key: "068705f79343a0aa5bb6bf76c6a37057"
server_key_iv: "f1fccb54cdc37dc1fbde91caad77324d"
server_key_version: "encryptionkey@0"
I20260924 16:10:59.457796 23073 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 16:10:59.462486 23087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:59.464517 23073 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:10:59.464766 23073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "acfd131ec733487d9ab8894f2ed5f62a"
format_stamp: "Formatted at 2026-09-24 16:10:59 on dist-test-slave-1gc3"
server_key: "068705f79343a0aa5bb6bf76c6a37057"
server_key_iv: "f1fccb54cdc37dc1fbde91caad77324d"
server_key_version: "encryptionkey@0"
I20260924 16:10:59.465037 23073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:10:59.486385 23073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:10:59.487313 23073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:10:59.487619 23073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:10:59.488783 23073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:10:59.490571 23073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:10:59.490676 23073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:59.490789 23073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:10:59.490859 23073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:59.527236 23073 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:37589
I20260924 16:10:59.527374 23199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:37589 every 8 connection(s)
I20260924 16:10:59.528697 23073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:10:59.536731 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 23073
I20260924 16:10:59.536974 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:10:59.537554 3931 external_mini_cluster.cc:1428] Setting key 2cad2fddb9698a80719c955cec895a7d
I20260924 16:10:59.541159 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:36393
--builtin_ntp_servers=127.3.214.212:37471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:10:59.543824 23202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:59.532197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:59577 (local address 127.3.214.254:36393)
0924 16:10:59.532762 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0924 16:10:59.532767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:10:59.534523 (+ 1756us) server_negotiation.cc:407] Connection header received
0924 16:10:59.536529 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:10:59.536535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:10:59.536625 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:10:59.536723 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:10:59.538155 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:59.538959 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:59.540623 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:10:59.541045 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:10:59.541161 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:59.542883 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:10:59.542890 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:10:59.542895 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:10:59.542920 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:10:59.542970 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:10:59.542985 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:10:59.543072 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 16:10:59.543327 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 16:10:59.543459 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":314,"threads_started":1}
I20260924 16:10:59.545040 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:10:59.532923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:36393 (local address 127.3.214.193:59577)
0924 16:10:59.534253 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0924 16:10:59.534337 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0924 16:10:59.536186 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:10:59.537030 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:10:59.537039 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:10:59.537162 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:10:59.537873 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:59.537903 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:59.539133 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:59.539137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:59.540459 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:10:59.540474 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:10:59.541294 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:10:59.541302 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:10:59.541577 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:10:59.542468 (+ 891us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:10:59.542471 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:10:59.542518 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:10:59.542633 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:10:59.543094 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:10:59.543107 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:10:59.543114 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 16:10:59.543299 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 16:10:59.543797 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":399,"threads_started":1}
I20260924 16:10:59.547340 23200 heartbeater.cc:347] Connected to a master server at 127.3.214.254:36393
I20260924 16:10:59.547961 23200 heartbeater.cc:464] Registering TS with master...
I20260924 16:10:59.549371 23200 heartbeater.cc:511] Master 127.3.214.254:36393 requested a full tablet report, sending...
I20260924 16:10:59.553076 23013 ts_manager.cc:194] Registered new tserver with Master: acfd131ec733487d9ab8894f2ed5f62a (127.3.214.193:37589)
I20260924 16:10:59.555271 23013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.214.193:59577
W20260924 16:10:59.929039 23204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:10:59.929450 23204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:10:59.929577 23204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:10:59.937950 23204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:10:59.938146 23204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:10:59.938324 23204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:10:59.949103 23204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:37471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:36393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:10:59.950991 23204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:10:59.953276 23204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:10:59.969620 23209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:59.969511 23212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:10:59.970010 23210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:10:59.970597 23204 server_base.cc:1034] running on GCE node
I20260924 16:10:59.971375 23204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:10:59.972617 23204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:10:59.973896 23204 hybrid_clock.cc:648] HybridClock initialized: now 1790266259973798 us; error 84 us; skew 500 ppm
I20260924 16:10:59.974416 23204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:10:59.977914 23204 webserver.cc:460] Webserver started at http://127.3.214.194:43511/ using document root <none> and password file <none>
I20260924 16:10:59.978909 23204 fs_manager.cc:359] Metadata directory not provided
I20260924 16:10:59.979048 23204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:10:59.979403 23204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:10:59.983186 23204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "500ab0a6e85b4551aeb13bc7eb6a1871"
format_stamp: "Formatted at 2026-09-24 16:10:59 on dist-test-slave-1gc3"
server_key: "4a8e33de3ab3a14bc22e1fb7426339ce"
server_key_iv: "161553b3e18f28876f3b2648a69e59b2"
server_key_version: "encryptionkey@0"
I20260924 16:10:59.984170 23204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "500ab0a6e85b4551aeb13bc7eb6a1871"
format_stamp: "Formatted at 2026-09-24 16:10:59 on dist-test-slave-1gc3"
server_key: "4a8e33de3ab3a14bc22e1fb7426339ce"
server_key_iv: "161553b3e18f28876f3b2648a69e59b2"
server_key_version: "encryptionkey@0"
I20260924 16:10:59.990762 23204 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 16:10:59.995543 23218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:10:59.997783 23204 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:10:59.998163 23204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "500ab0a6e85b4551aeb13bc7eb6a1871"
format_stamp: "Formatted at 2026-09-24 16:10:59 on dist-test-slave-1gc3"
server_key: "4a8e33de3ab3a14bc22e1fb7426339ce"
server_key_iv: "161553b3e18f28876f3b2648a69e59b2"
server_key_version: "encryptionkey@0"
I20260924 16:10:59.998998 23204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:00.017168 23204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:00.018217 23204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:00.018554 23204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:00.019790 23204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:11:00.022064 23204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:00.022228 23204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 16:11:00.022363 23204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:00.022446 23204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:00.059574 23204 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:45575
I20260924 16:11:00.059665 23330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:45575 every 8 connection(s)
I20260924 16:11:00.061074 23204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:11:00.068387 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 23204
I20260924 16:11:00.068614 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:11:00.069237 3931 external_mini_cluster.cc:1428] Setting key 60a419f410998b61e804359d684913e4
I20260924 16:11:00.073240 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:36393
--builtin_ntp_servers=127.3.214.212:37471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:11:00.079737 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:00.064717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:51593 (local address 127.3.214.254:36393)
0924 16:11:00.065344 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:00.065352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:00.067128 (+ 1776us) server_negotiation.cc:407] Connection header received
0924 16:11:00.070195 (+ 3067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:00.070203 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:00.070320 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:00.070460 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:00.072217 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.073094 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.075864 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.076369 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.076528 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:00.078738 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:00.078744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:00.078748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:00.078776 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:00.078839 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:00.078859 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:00.078957 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:00.079310 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 16:11:00.079427 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":281,"threads_started":1}
I20260924 16:11:00.081605 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:00.065383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:36393 (local address 127.3.214.194:51593)
0924 16:11:00.066829 (+ 1446us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:00.066897 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:00.069514 (+ 2617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:00.070883 (+ 1369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:00.070896 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:00.071037 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:00.071987 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:00.072021 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.073315 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.073321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:00.075609 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:00.075630 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.076695 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.076701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:00.077068 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:00.078159 (+ 1091us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:11:00.078162 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:00.078235 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:11:00.078381 (+ 146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:00.078978 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:00.078991 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:00.079000 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 16:11:00.079228 (+ 228us) client_negotiation.cc:240] Negotiation successful
0924 16:11:00.079832 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":396,"threads_started":1}
I20260924 16:11:00.083837 23331 heartbeater.cc:347] Connected to a master server at 127.3.214.254:36393
I20260924 16:11:00.084312 23331 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:00.085528 23331 heartbeater.cc:511] Master 127.3.214.254:36393 requested a full tablet report, sending...
I20260924 16:11:00.087733 23013 ts_manager.cc:194] Registered new tserver with Master: 500ab0a6e85b4551aeb13bc7eb6a1871 (127.3.214.194:45575)
I20260924 16:11:00.088909 23013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.214.194:51593
W20260924 16:11:00.487224 23335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:00.487640 23335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:00.487775 23335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:00.496830 23335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:00.497004 23335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:00.497169 23335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:11:00.508697 23335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:37471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:36393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:00.510732 23335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:00.513237 23335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:00.529340 23341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:00.529992 23343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:00.530658 23340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:00.532807 23335 server_base.cc:1034] running on GCE node
I20260924 16:11:00.533618 23335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:00.534935 23335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:00.536243 23335 hybrid_clock.cc:648] HybridClock initialized: now 1790266260536153 us; error 88 us; skew 500 ppm
I20260924 16:11:00.536667 23335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:11:00.539413 23335 webserver.cc:460] Webserver started at http://127.3.214.195:43557/ using document root <none> and password file <none>
I20260924 16:11:00.540349 23335 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:00.540462 23335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:00.540827 23335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:00.544095 23335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "a0e009b126c94519bd7812f64e921b00"
format_stamp: "Formatted at 2026-09-24 16:11:00 on dist-test-slave-1gc3"
server_key: "af199dde80943c8c2c7238a7db3f44dc"
server_key_iv: "e1c63fc81dca43853d374da85dd488fd"
server_key_version: "encryptionkey@0"
I20260924 16:11:00.544958 23335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "a0e009b126c94519bd7812f64e921b00"
format_stamp: "Formatted at 2026-09-24 16:11:00 on dist-test-slave-1gc3"
server_key: "af199dde80943c8c2c7238a7db3f44dc"
server_key_iv: "e1c63fc81dca43853d374da85dd488fd"
server_key_version: "encryptionkey@0"
I20260924 16:11:00.551766 23335 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 16:11:00.556385 23349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:00.558212 23335 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:11:00.558439 23335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "a0e009b126c94519bd7812f64e921b00"
format_stamp: "Formatted at 2026-09-24 16:11:00 on dist-test-slave-1gc3"
server_key: "af199dde80943c8c2c7238a7db3f44dc"
server_key_iv: "e1c63fc81dca43853d374da85dd488fd"
server_key_version: "encryptionkey@0"
I20260924 16:11:00.558717 23335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:00.560184 23200 heartbeater.cc:503] Master 127.3.214.254:36393 was elected leader, sending a full tablet report...
I20260924 16:11:00.577818 23335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:00.578894 23335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:00.579200 23335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:00.580472 23335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:11:00.582549 23335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:00.582666 23335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:00.582837 23335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:00.582928 23335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:00.625255 23335 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:35515
I20260924 16:11:00.625373 23461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:35515 every 8 connection(s)
I20260924 16:11:00.626785 23335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:11:00.632299 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 23335
I20260924 16:11:00.632535 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:11:00.633183 3931 external_mini_cluster.cc:1428] Setting key 8533b7f4aabe16a60658128df1156ef6
I20260924 16:11:00.643231 23464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:00.630323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:41037 (local address 127.3.214.254:36393)
0924 16:11:00.630842 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:00.630847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:00.632663 (+ 1816us) server_negotiation.cc:407] Connection header received
0924 16:11:00.635228 (+ 2565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:00.635233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:00.635332 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:00.635451 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:00.637019 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.637808 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.639706 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.640154 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.640275 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:00.642014 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:00.642020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:00.642024 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:00.642056 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:00.642157 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:00.642185 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:00.642290 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:00.642651 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 16:11:00.642786 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":265,"threads_started":1}
I20260924 16:11:00.644379 23465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:00.630854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:36393 (local address 127.3.214.195:41037)
0924 16:11:00.632267 (+ 1413us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:00.632344 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:00.634853 (+ 2509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:00.635784 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:00.635799 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:00.635954 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:00.636837 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:00.636862 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.637975 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.637979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:00.639533 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:00.639547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.640447 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.640455 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:00.640742 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:00.641538 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:11:00.641541 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:00.641605 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:11:00.641719 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:00.642330 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:00.642341 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:00.642348 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 16:11:00.642557 (+ 209us) client_negotiation.cc:240] Negotiation successful
0924 16:11:00.642976 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":383,"threads_started":1}
I20260924 16:11:00.646230 23462 heartbeater.cc:347] Connected to a master server at 127.3.214.254:36393
I20260924 16:11:00.646579 23462 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:00.647667 23462 heartbeater.cc:511] Master 127.3.214.254:36393 requested a full tablet report, sending...
I20260924 16:11:00.649768 23013 ts_manager.cc:194] Registered new tserver with Master: a0e009b126c94519bd7812f64e921b00 (127.3.214.195:35515)
I20260924 16:11:00.650514 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:11:00.650853 23013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.214.195:41037
I20260924 16:11:00.663139 23464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:00.656405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41696 (local address 127.3.214.254:36393)
0924 16:11:00.656641 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:00.656649 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:00.657078 (+ 429us) server_negotiation.cc:407] Connection header received
0924 16:11:00.657313 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:00.657318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:00.657392 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:00.657482 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:00.657988 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.658848 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.660300 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.660759 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.660873 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:00.662319 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:00.662324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:00.662328 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:00.662350 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:00.662389 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:00.662402 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:00.662535 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:00.662755 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 16:11:00.662825 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 16:11:00.668701 23013 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:11:00.676433 23013 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:11:00.709100 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:00.694273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:37589 (local address 127.0.0.1:34658)
0924 16:11:00.696552 (+ 2279us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:00.696576 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:00.696691 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:00.697716 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:00.697725 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:00.697761 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:00.699131 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:00.699159 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.701732 (+ 2573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.701742 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:00.705312 (+ 3570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:00.705353 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.707956 (+ 2603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.707963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:00.708173 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:00.708246 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 16:11:00.708395 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 16:11:00.708652 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2141,"mutex_wait_us":33,"thread_start_us":1034,"threads_started":1}
I20260924 16:11:00.710158 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:00.694463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34658 (local address 127.3.214.193:37589)
0924 16:11:00.696862 (+ 2399us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:00.696875 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:00.696920 (+ 45us) server_negotiation.cc:407] Connection header received
0924 16:11:00.697056 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:00.697065 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:00.697405 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:00.698422 (+ 1017us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:00.699445 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.701460 (+ 2015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.706262 (+ 4802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.707740 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.707878 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:00.708236 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:00.709031 (+ 795us) server_negotiation.cc:299] Negotiation successful
0924 16:11:00.709690 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2163,"thread_start_us":1980,"threads_started":1}
I20260924 16:11:00.711138 23478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:00.695101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35728 (local address 127.3.214.194:45575)
0924 16:11:00.697282 (+ 2181us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:00.697290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:00.697321 (+ 31us) server_negotiation.cc:407] Connection header received
0924 16:11:00.697432 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:00.697439 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:00.697645 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:00.697847 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:00.699058 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.701740 (+ 2682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.706468 (+ 4728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.708304 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.708477 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:00.708769 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:00.710583 (+ 1814us) server_negotiation.cc:299] Negotiation successful
0924 16:11:00.710789 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1993,"thread_start_us":1825,"threads_started":1}
I20260924 16:11:00.711514 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:00.694187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:45575 (local address 127.0.0.1:35728)
0924 16:11:00.695841 (+ 1654us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:00.695891 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:00.696109 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:00.698147 (+ 2038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:00.698153 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:00.698181 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:00.698772 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:00.698791 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.702056 (+ 3265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.702064 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:00.706145 (+ 4081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:00.706166 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.708978 (+ 2812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.708986 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:00.709610 (+ 624us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:00.709653 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 16:11:00.709744 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 16:11:00.709909 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1301,"mutex_wait_us":6,"thread_start_us":2045,"threads_started":1}
I20260924 16:11:00.715679 23135 tablet_service.cc:1467] Processing CreateTablet for tablet ba1af76091bb41df92da99a8192fe878 (DEFAULT_TABLE table=test-table [id=ffa17e605e3a42eca61c2ef21c7afed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:00.716723 23266 tablet_service.cc:1467] Processing CreateTablet for tablet ba1af76091bb41df92da99a8192fe878 (DEFAULT_TABLE table=test-table [id=ffa17e605e3a42eca61c2ef21c7afed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:00.717698 23135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1af76091bb41df92da99a8192fe878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:00.718441 23479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:00.697716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:35515 (local address 127.0.0.1:34168)
0924 16:11:00.705814 (+ 8098us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:00.705838 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:00.706035 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:00.710092 (+ 4057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:00.710123 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:00.710157 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:00.710591 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:00.710605 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.712587 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.712594 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:00.714927 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:00.714959 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.717239 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.717247 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:00.717472 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:00.717693 (+ 221us) client_negotiation.cc:769] Sending connection context
0924 16:11:00.717797 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 16:11:00.717996 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7919,"thread_start_us":7682,"threads_started":1}
I20260924 16:11:00.718652 23266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1af76091bb41df92da99a8192fe878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:00.719098 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:00.698406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34168 (local address 127.3.214.195:35515)
0924 16:11:00.708931 (+ 10525us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:00.708942 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:00.708988 (+ 46us) server_negotiation.cc:407] Connection header received
0924 16:11:00.709125 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:00.709134 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:00.709491 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:00.709739 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:00.710823 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.712369 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.715330 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.716905 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.717115 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:00.717461 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:00.718054 (+ 593us) server_negotiation.cc:299] Negotiation successful
0924 16:11:00.718294 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10268,"thread_start_us":10078,"threads_started":1}
I20260924 16:11:00.723801 23397 tablet_service.cc:1467] Processing CreateTablet for tablet ba1af76091bb41df92da99a8192fe878 (DEFAULT_TABLE table=test-table [id=ffa17e605e3a42eca61c2ef21c7afed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:00.725826 23397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1af76091bb41df92da99a8192fe878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:00.733443 23481 tablet_bootstrap.cc:492] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871: Bootstrap starting.
I20260924 16:11:00.736192 23482 tablet_bootstrap.cc:492] T ba1af76091bb41df92da99a8192fe878 P acfd131ec733487d9ab8894f2ed5f62a: Bootstrap starting.
I20260924 16:11:00.739879 23482 tablet_bootstrap.cc:654] T ba1af76091bb41df92da99a8192fe878 P acfd131ec733487d9ab8894f2ed5f62a: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:00.739921 23481 tablet_bootstrap.cc:654] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:00.741200 23482 log.cc:824] T ba1af76091bb41df92da99a8192fe878 P acfd131ec733487d9ab8894f2ed5f62a: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:00.741446 23481 log.cc:824] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:00.743517 23484 tablet_bootstrap.cc:492] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00: Bootstrap starting.
I20260924 16:11:00.745205 23482 tablet_bootstrap.cc:492] T ba1af76091bb41df92da99a8192fe878 P acfd131ec733487d9ab8894f2ed5f62a: No bootstrap required, opened a new log
I20260924 16:11:00.745568 23482 ts_tablet_manager.cc:1397] T ba1af76091bb41df92da99a8192fe878 P acfd131ec733487d9ab8894f2ed5f62a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 16:11:00.746428 23481 tablet_bootstrap.cc:492] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871: No bootstrap required, opened a new log
I20260924 16:11:00.746730 23481 ts_tablet_manager.cc:1397] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20260924 16:11:00.747359 23484 tablet_bootstrap.cc:654] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:00.748849 23484 log.cc:824] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:00.752063 23484 tablet_bootstrap.cc:492] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00: No bootstrap required, opened a new log
I20260924 16:11:00.752372 23484 ts_tablet_manager.cc:1397] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 16:11:00.757570 23481 raft_consensus.cc:348] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfd131ec733487d9ab8894f2ed5f62a" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37589 } } peers { permanent_uuid: "500ab0a6e85b4551aeb13bc7eb6a1871" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45575 } } peers { permanent_uuid: "a0e009b126c94519bd7812f64e921b00" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35515 } }
I20260924 16:11:00.757522 23482 raft_consensus.cc:348] T ba1af76091bb41df92da99a8192fe878 P acfd131ec733487d9ab8894f2ed5f62a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfd131ec733487d9ab8894f2ed5f62a" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37589 } } peers { permanent_uuid: "500ab0a6e85b4551aeb13bc7eb6a1871" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45575 } } peers { permanent_uuid: "a0e009b126c94519bd7812f64e921b00" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35515 } }
I20260924 16:11:00.758095 23482 raft_consensus.cc:374] T ba1af76091bb41df92da99a8192fe878 P acfd131ec733487d9ab8894f2ed5f62a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:00.758093 23481 raft_consensus.cc:374] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:00.758236 23482 raft_consensus.cc:729] T ba1af76091bb41df92da99a8192fe878 P acfd131ec733487d9ab8894f2ed5f62a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acfd131ec733487d9ab8894f2ed5f62a, State: Initialized, Role: FOLLOWER
I20260924 16:11:00.758236 23481 raft_consensus.cc:729] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500ab0a6e85b4551aeb13bc7eb6a1871, State: Initialized, Role: FOLLOWER
I20260924 16:11:00.758961 23482 consensus_queue.cc:260] T ba1af76091bb41df92da99a8192fe878 P acfd131ec733487d9ab8894f2ed5f62a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfd131ec733487d9ab8894f2ed5f62a" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37589 } } peers { permanent_uuid: "500ab0a6e85b4551aeb13bc7eb6a1871" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45575 } } peers { permanent_uuid: "a0e009b126c94519bd7812f64e921b00" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35515 } }
I20260924 16:11:00.758965 23481 consensus_queue.cc:260] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfd131ec733487d9ab8894f2ed5f62a" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37589 } } peers { permanent_uuid: "500ab0a6e85b4551aeb13bc7eb6a1871" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45575 } } peers { permanent_uuid: "a0e009b126c94519bd7812f64e921b00" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35515 } }
I20260924 16:11:00.761098 23331 heartbeater.cc:503] Master 127.3.214.254:36393 was elected leader, sending a full tablet report...
I20260924 16:11:00.761518 23481 ts_tablet_manager.cc:1428] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260924 16:11:00.762944 23482 ts_tablet_manager.cc:1428] T ba1af76091bb41df92da99a8192fe878 P acfd131ec733487d9ab8894f2ed5f62a: Time spent starting tablet: real 0.017s user 0.013s sys 0.005s
I20260924 16:11:00.762832 23484 raft_consensus.cc:348] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfd131ec733487d9ab8894f2ed5f62a" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37589 } } peers { permanent_uuid: "500ab0a6e85b4551aeb13bc7eb6a1871" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45575 } } peers { permanent_uuid: "a0e009b126c94519bd7812f64e921b00" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35515 } }
I20260924 16:11:00.763453 23484 raft_consensus.cc:374] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:00.763577 23484 raft_consensus.cc:729] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e009b126c94519bd7812f64e921b00, State: Initialized, Role: FOLLOWER
I20260924 16:11:00.764421 23484 consensus_queue.cc:260] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfd131ec733487d9ab8894f2ed5f62a" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37589 } } peers { permanent_uuid: "500ab0a6e85b4551aeb13bc7eb6a1871" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45575 } } peers { permanent_uuid: "a0e009b126c94519bd7812f64e921b00" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35515 } }
I20260924 16:11:00.765954 23462 heartbeater.cc:503] Master 127.3.214.254:36393 was elected leader, sending a full tablet report...
I20260924 16:11:00.766728 23484 ts_tablet_manager.cc:1428] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260924 16:11:00.781653 23201 tablet.cc:2369] T ba1af76091bb41df92da99a8192fe878 P acfd131ec733487d9ab8894f2ed5f62a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:11:00.813823 23332 tablet.cc:2369] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:11:00.840397 23489 raft_consensus.cc:482] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:11:00.840840 23489 raft_consensus.cc:504] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfd131ec733487d9ab8894f2ed5f62a" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37589 } } peers { permanent_uuid: "500ab0a6e85b4551aeb13bc7eb6a1871" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45575 } } peers { permanent_uuid: "a0e009b126c94519bd7812f64e921b00" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35515 } }
I20260924 16:11:00.843194 23489 leader_election.cc:288] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acfd131ec733487d9ab8894f2ed5f62a (127.3.214.193:37589), 500ab0a6e85b4551aeb13bc7eb6a1871 (127.3.214.194:45575)
I20260924 16:11:00.849825 23488 raft_consensus.cc:482] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:11:00.850304 23488 raft_consensus.cc:504] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfd131ec733487d9ab8894f2ed5f62a" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37589 } } peers { permanent_uuid: "500ab0a6e85b4551aeb13bc7eb6a1871" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45575 } } peers { permanent_uuid: "a0e009b126c94519bd7812f64e921b00" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35515 } }
I20260924 16:11:00.851164 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:00.843713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:34585 (local address 127.3.214.193:37589)
0924 16:11:00.844137 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:00.844146 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:00.844186 (+ 40us) server_negotiation.cc:407] Connection header received
0924 16:11:00.844406 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:00.844413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:00.844529 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:00.844661 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:00.845124 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.846227 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.848725 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.850076 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.850286 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:00.850370 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:00.850702 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 16:11:00.850854 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260924 16:11:00.851192 23465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:00.843530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:37589 (local address 127.3.214.195:34585)
0924 16:11:00.843970 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:00.844000 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:00.844156 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:00.844727 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:00.844731 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:00.844758 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:00.844957 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:00.844966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.846409 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.846415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:00.848492 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:00.848523 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.850379 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.850385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:00.850535 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:00.850563 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 16:11:00.850660 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 16:11:00.850855 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":110}
I20260924 16:11:00.852445 23490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:00.843863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:45575 (local address 127.3.214.195:50067)
0924 16:11:00.844832 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:00.844852 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:00.844996 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:00.845435 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:00.845440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:00.845470 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:00.845903 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:00.845916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.847521 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.847527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:00.849708 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:00.849727 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.851647 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.851654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:00.851817 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:00.851843 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 16:11:00.851930 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 16:11:00.852072 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"mutex_wait_us":160,"thread_start_us":494,"threads_started":1}
I20260924 16:11:00.852833 23478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:00.844034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:50067 (local address 127.3.214.194:45575)
0924 16:11:00.844262 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:00.844269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:00.844944 (+ 675us) server_negotiation.cc:407] Connection header received
0924 16:11:00.845206 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:00.845212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:00.845304 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:00.845382 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:00.846158 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.847281 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.850033 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.851312 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.851541 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:00.851621 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:00.852013 (+ 392us) server_negotiation.cc:299] Negotiation successful
0924 16:11:00.852212 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 16:11:00.852717 23155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1af76091bb41df92da99a8192fe878" candidate_uuid: "a0e009b126c94519bd7812f64e921b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acfd131ec733487d9ab8894f2ed5f62a" is_pre_election: true
I20260924 16:11:00.853381 23155 raft_consensus.cc:2454] T ba1af76091bb41df92da99a8192fe878 P acfd131ec733487d9ab8894f2ed5f62a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e009b126c94519bd7812f64e921b00 in term 0.
I20260924 16:11:00.852842 23488 leader_election.cc:288] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acfd131ec733487d9ab8894f2ed5f62a (127.3.214.193:37589), a0e009b126c94519bd7812f64e921b00 (127.3.214.195:35515)
I20260924 16:11:00.854489 23351 leader_election.cc:302] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0e009b126c94519bd7812f64e921b00, acfd131ec733487d9ab8894f2ed5f62a; no voters:
I20260924 16:11:00.855111 23489 raft_consensus.cc:2781] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:11:00.855275 23489 raft_consensus.cc:482] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:11:00.855393 23489 raft_consensus.cc:3037] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:00.856230 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1af76091bb41df92da99a8192fe878" candidate_uuid: "a0e009b126c94519bd7812f64e921b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "500ab0a6e85b4551aeb13bc7eb6a1871" is_pre_election: true
I20260924 16:11:00.856835 23286 raft_consensus.cc:2454] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e009b126c94519bd7812f64e921b00 in term 0.
I20260924 16:11:00.858811 23489 raft_consensus.cc:504] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfd131ec733487d9ab8894f2ed5f62a" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37589 } } peers { permanent_uuid: "500ab0a6e85b4551aeb13bc7eb6a1871" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45575 } } peers { permanent_uuid: "a0e009b126c94519bd7812f64e921b00" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35515 } }
I20260924 16:11:00.860404 23155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1af76091bb41df92da99a8192fe878" candidate_uuid: "a0e009b126c94519bd7812f64e921b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acfd131ec733487d9ab8894f2ed5f62a"
I20260924 16:11:00.860402 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1af76091bb41df92da99a8192fe878" candidate_uuid: "a0e009b126c94519bd7812f64e921b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "500ab0a6e85b4551aeb13bc7eb6a1871"
I20260924 16:11:00.860671 23155 raft_consensus.cc:3037] T ba1af76091bb41df92da99a8192fe878 P acfd131ec733487d9ab8894f2ed5f62a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:00.860759 23286 raft_consensus.cc:3037] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:00.864396 23155 raft_consensus.cc:2454] T ba1af76091bb41df92da99a8192fe878 P acfd131ec733487d9ab8894f2ed5f62a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e009b126c94519bd7812f64e921b00 in term 1.
I20260924 16:11:00.864754 23286 raft_consensus.cc:2454] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e009b126c94519bd7812f64e921b00 in term 1.
I20260924 16:11:00.864859 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:00.855981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:56473 (local address 127.3.214.193:37589)
0924 16:11:00.856198 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:00.856203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:00.856220 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:11:00.856306 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:00.856311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:00.856393 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:00.856472 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:00.857829 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.859049 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.862151 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.863560 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.863716 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:00.863797 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:00.864378 (+ 581us) server_negotiation.cc:299] Negotiation successful
0924 16:11:00.864543 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 16:11:00.865008 23351 leader_election.cc:302] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0e009b126c94519bd7812f64e921b00, acfd131ec733487d9ab8894f2ed5f62a; no voters:
I20260924 16:11:00.865625 23489 leader_election.cc:288] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00 [CANDIDATE]: Term 1 election: Requested vote from peers acfd131ec733487d9ab8894f2ed5f62a (127.3.214.193:37589), 500ab0a6e85b4551aeb13bc7eb6a1871 (127.3.214.194:45575)
I20260924 16:11:00.865694 23491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:00.853123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:37589 (local address 127.3.214.194:56473)
0924 16:11:00.855441 (+ 2318us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:00.855517 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:00.855685 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:00.856745 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:00.856750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:00.856784 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:00.857634 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:00.857650 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.859247 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.859252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:00.861857 (+ 2605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:00.861874 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.863988 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.863995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:00.864183 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:00.864218 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 16:11:00.865108 (+ 890us) client_negotiation.cc:240] Negotiation successful
0924 16:11:00.865285 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2026,"mutex_wait_us":51,"thread_start_us":1752,"threads_started":1}
I20260924 16:11:00.865892 23489 raft_consensus.cc:2781] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:00.866318 23489 raft_consensus.cc:686] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00 [term 1 LEADER]: Becoming Leader. State: Replica: a0e009b126c94519bd7812f64e921b00, State: Running, Role: LEADER
I20260924 16:11:00.866765 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:00.853716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:48647 (local address 127.3.214.195:35515)
0924 16:11:00.853965 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:00.853972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:00.856245 (+ 2273us) server_negotiation.cc:407] Connection header received
0924 16:11:00.856536 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:00.856541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:00.856622 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:00.856706 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:00.857997 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.859177 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.863432 (+ 4255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.864654 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.864823 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:00.864902 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:00.866297 (+ 1395us) server_negotiation.cc:299] Negotiation successful
0924 16:11:00.866490 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 16:11:00.866591 23154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1af76091bb41df92da99a8192fe878" candidate_uuid: "500ab0a6e85b4551aeb13bc7eb6a1871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acfd131ec733487d9ab8894f2ed5f62a" is_pre_election: true
I20260924 16:11:00.867060 23154 raft_consensus.cc:2379] T ba1af76091bb41df92da99a8192fe878 P acfd131ec733487d9ab8894f2ed5f62a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 500ab0a6e85b4551aeb13bc7eb6a1871 in current term 1: Already voted for candidate a0e009b126c94519bd7812f64e921b00 in this term.
I20260924 16:11:00.867154 23492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:00.853485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:35515 (local address 127.3.214.194:48647)
0924 16:11:00.856136 (+ 2651us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:00.856154 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:00.856345 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:00.856887 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:00.856891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:00.856937 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:00.857794 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:00.857817 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.859353 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.859367 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:00.862162 (+ 2795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:00.862187 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.865427 (+ 3240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.865436 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:00.865648 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:00.865694 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 16:11:00.866236 (+ 542us) client_negotiation.cc:240] Negotiation successful
0924 16:11:00.866504 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2516,"mutex_wait_us":101,"thread_start_us":3665,"threads_started":1}
I20260924 16:11:00.867179 23489 consensus_queue.cc:237] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfd131ec733487d9ab8894f2ed5f62a" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37589 } } peers { permanent_uuid: "500ab0a6e85b4551aeb13bc7eb6a1871" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45575 } } peers { permanent_uuid: "a0e009b126c94519bd7812f64e921b00" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35515 } }
I20260924 16:11:00.868295 23417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1af76091bb41df92da99a8192fe878" candidate_uuid: "500ab0a6e85b4551aeb13bc7eb6a1871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e009b126c94519bd7812f64e921b00" is_pre_election: true
I20260924 16:11:00.869652 23222 leader_election.cc:302] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 500ab0a6e85b4551aeb13bc7eb6a1871; no voters: a0e009b126c94519bd7812f64e921b00, acfd131ec733487d9ab8894f2ed5f62a
I20260924 16:11:00.870056 23488 raft_consensus.cc:2726] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 16:11:00.876400 23012 catalog_manager.cc:5495] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0e009b126c94519bd7812f64e921b00 (127.3.214.195). New cstate: current_term: 1 leader_uuid: "a0e009b126c94519bd7812f64e921b00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acfd131ec733487d9ab8894f2ed5f62a" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "500ab0a6e85b4551aeb13bc7eb6a1871" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0e009b126c94519bd7812f64e921b00" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 35515 } health_report { overall_health: HEALTHY } } }
W20260924 16:11:00.878895 23463 tablet.cc:2369] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:11:00.923592 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:00.916767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34180 (local address 127.3.214.195:35515)
0924 16:11:00.917060 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:00.917066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:00.917096 (+ 30us) server_negotiation.cc:407] Connection header received
0924 16:11:00.917355 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:00.917360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:00.917514 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:00.917702 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:00.918307 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.919172 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.920708 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:00.921197 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:00.921307 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:00.921921 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:00.922671 (+ 750us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:00.922827 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:00.923080 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 16:11:00.923160 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260924 16:11:00.934053 23155 raft_consensus.cc:1261] T ba1af76091bb41df92da99a8192fe878 P acfd131ec733487d9ab8894f2ed5f62a [term 1 FOLLOWER]: Refusing update from remote peer a0e009b126c94519bd7812f64e921b00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:00.934206 23286 raft_consensus.cc:1261] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871 [term 1 FOLLOWER]: Refusing update from remote peer a0e009b126c94519bd7812f64e921b00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:00.935148 23493 consensus_queue.cc:1035] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acfd131ec733487d9ab8894f2ed5f62a" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 37589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:11:00.935679 23489 consensus_queue.cc:1035] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "500ab0a6e85b4551aeb13bc7eb6a1871" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 45575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:11:00.969308 23013 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:11:00.969699 23013 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:11:00.974749 23013 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15d7ab100e624b68a3cc492edd012e0e: Sending DeleteTablet for 3 replicas of tablet ba1af76091bb41df92da99a8192fe878
I20260924 16:11:00.976346 23266 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1af76091bb41df92da99a8192fe878 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:00 UTC) from {username='slave'} at 127.0.0.1:35728
I20260924 16:11:00.976346 23135 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1af76091bb41df92da99a8192fe878 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:00 UTC) from {username='slave'} at 127.0.0.1:34658
I20260924 16:11:00.977012 23397 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1af76091bb41df92da99a8192fe878 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:00 UTC) from {username='slave'} at 127.0.0.1:34168
I20260924 16:11:00.981189 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 23073
I20260924 16:11:00.989283 23507 tablet_replica.cc:330] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871: stopping tablet replica
I20260924 16:11:00.990314 23507 raft_consensus.cc:2229] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:11:00.991259 23507 raft_consensus.cc:2258] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:00.992399 23506 tablet_replica.cc:330] T ba1af76091bb41df92da99a8192fe878 P acfd131ec733487d9ab8894f2ed5f62a: stopping tablet replica
I20260924 16:11:00.992853 23506 raft_consensus.cc:2229] T ba1af76091bb41df92da99a8192fe878 P acfd131ec733487d9ab8894f2ed5f62a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:11:00.993325 23506 raft_consensus.cc:2258] T ba1af76091bb41df92da99a8192fe878 P acfd131ec733487d9ab8894f2ed5f62a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:00.995908 23508 tablet_replica.cc:330] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00: stopping tablet replica
I20260924 16:11:00.996428 23506 ts_tablet_manager.cc:1905] T ba1af76091bb41df92da99a8192fe878 P acfd131ec733487d9ab8894f2ed5f62a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:00.996577 23508 raft_consensus.cc:2229] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:11:00.997596 23508 raft_consensus.cc:2258] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:00.998656 23507 ts_tablet_manager.cc:1905] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:01.000108 23508 ts_tablet_manager.cc:1905] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:01.005812 23506 ts_tablet_manager.cc:1918] T ba1af76091bb41df92da99a8192fe878 P acfd131ec733487d9ab8894f2ed5f62a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:11:01.006072 23506 log.cc:1196] T ba1af76091bb41df92da99a8192fe878 P acfd131ec733487d9ab8894f2ed5f62a: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-0/wal/wals/ba1af76091bb41df92da99a8192fe878
I20260924 16:11:01.006680 23506 ts_tablet_manager.cc:1939] T ba1af76091bb41df92da99a8192fe878 P acfd131ec733487d9ab8894f2ed5f62a: Deleting consensus metadata
I20260924 16:11:01.009435 22998 catalog_manager.cc:4841] TS acfd131ec733487d9ab8894f2ed5f62a (127.3.214.193:37589): tablet ba1af76091bb41df92da99a8192fe878 (table test-table [id=ffa17e605e3a42eca61c2ef21c7afed9]) successfully deleted
I20260924 16:11:01.010241 23508 ts_tablet_manager.cc:1918] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:11:01.010478 23508 log.cc:1196] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/ba1af76091bb41df92da99a8192fe878
I20260924 16:11:01.011005 23508 ts_tablet_manager.cc:1939] T ba1af76091bb41df92da99a8192fe878 P a0e009b126c94519bd7812f64e921b00: Deleting consensus metadata
I20260924 16:11:01.013432 23000 catalog_manager.cc:4841] TS a0e009b126c94519bd7812f64e921b00 (127.3.214.195:35515): tablet ba1af76091bb41df92da99a8192fe878 (table test-table [id=ffa17e605e3a42eca61c2ef21c7afed9]) successfully deleted
I20260924 16:11:01.014300 23507 ts_tablet_manager.cc:1918] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:11:01.014819 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:00.983027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34674 (local address 127.3.214.193:37589)
0924 16:11:00.983259 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:00.983265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:00.994996 (+ 11731us) server_negotiation.cc:407] Connection header received
0924 16:11:00.995253 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:00.995260 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:00.995413 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:00.995580 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:01.000719 (+ 5139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:01.001975 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:01.004574 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:01.005256 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:01.005390 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:01.012579 (+ 7189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:01.012596 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:01.012621 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:01.013644 (+ 1023us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:01.013760 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:01.013798 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:01.013889 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:01.014278 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 16:11:01.014442 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 16:11:01.014519 23507 log.cc:1196] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/ba1af76091bb41df92da99a8192fe878
I20260924 16:11:01.015635 23507 ts_tablet_manager.cc:1939] T ba1af76091bb41df92da99a8192fe878 P 500ab0a6e85b4551aeb13bc7eb6a1871: Deleting consensus metadata
I20260924 16:11:01.015861 23195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:01.018925 22997 catalog_manager.cc:4841] TS 500ab0a6e85b4551aeb13bc7eb6a1871 (127.3.214.194:45575): tablet ba1af76091bb41df92da99a8192fe878 (table test-table [id=ffa17e605e3a42eca61c2ef21c7afed9]) successfully deleted
I20260924 16:11:01.121939 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 23073
I20260924 16:11:01.146230 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 23204
I20260924 16:11:01.155134 23478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:01.147209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35736 (local address 127.3.214.194:45575)
0924 16:11:01.147474 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:01.147481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:01.147552 (+ 71us) server_negotiation.cc:407] Connection header received
0924 16:11:01.147632 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:01.147638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:01.147781 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:01.147903 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:01.148370 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:01.149546 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:01.151380 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:01.152048 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:01.152193 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:01.153343 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:01.153355 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:01.153378 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:01.154232 (+ 854us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:01.154304 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:01.154333 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:01.154487 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:01.154638 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 16:11:01.154763 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260924 16:11:01.156040 23326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:01.263639 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 23204
I20260924 16:11:01.290248 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 23335
I20260924 16:11:01.299938 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:01.291455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34182 (local address 127.3.214.195:35515)
0924 16:11:01.291652 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:01.291657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:01.291680 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:11:01.291912 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:01.291917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:01.292080 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:01.292182 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:01.292864 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:01.293722 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:01.295895 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:01.296525 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:01.296713 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:01.298080 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:01.298094 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:01.298164 (+ 70us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:01.298994 (+ 830us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:01.299081 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:01.299112 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:01.299219 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:01.299411 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 16:11:01.299526 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:11:01.300833 23457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:01.433646 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 23335
I20260924 16:11:01.457178 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 22982
I20260924 16:11:01.458766 23044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:01.587581 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 22982
2026-09-24T16:11:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3372 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:01 dist-test-slave-1gc3 krb5kdc[23516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:01 dist-test-slave-1gc3 krb5kdc[23516](info): set up 2 sockets
Sep 24 16:11:01 dist-test-slave-1gc3 krb5kdc[23516](info): commencing operation
krb5kdc: starting...
W20260924 16:11:02.655956 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:11:02 dist-test-slave-1gc3 krb5kdc[23516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:11:02.968014 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:41577
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:41577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:03.351029 23529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:03.351369 23529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:03.351496 23529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:03.359284 23529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:03.359422 23529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:03.359478 23529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:03.359570 23529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:03.359622 23529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:03.370401 23529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:41577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:41577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:03.371953 23529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:03.373502 23529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:03.387677 23537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:03.387744 23534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:03.387768 23535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:03.389014 23529 server_base.cc:1034] running on GCE node
I20260924 16:11:03.389910 23529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:03.391605 23529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:03.392877 23529 hybrid_clock.cc:648] HybridClock initialized: now 1790266263392812 us; error 54 us; skew 500 ppm
I20260924 16:11:03.393352 23529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:03 dist-test-slave-1gc3 krb5kdc[23516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266263, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:03.397027 23529 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:11:03.398937 23529 webserver.cc:460] Webserver started at http://127.3.214.254:45457/ using document root <none> and password file <none>
I20260924 16:11:03.399684 23529 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:03.399819 23529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:03.400138 23529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:03.403430 23529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "e06bbf5ff8244f968785bf35811e7a66"
format_stamp: "Formatted at 2026-09-24 16:11:03 on dist-test-slave-1gc3"
server_key: "9ab6e1bd1407d71460abb8d8fd9e9cd5"
server_key_iv: "ab973c7f507d0e94e12524a116eab47e"
server_key_version: "encryptionkey@0"
I20260924 16:11:03.404215 23529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "e06bbf5ff8244f968785bf35811e7a66"
format_stamp: "Formatted at 2026-09-24 16:11:03 on dist-test-slave-1gc3"
server_key: "9ab6e1bd1407d71460abb8d8fd9e9cd5"
server_key_iv: "ab973c7f507d0e94e12524a116eab47e"
server_key_version: "encryptionkey@0"
I20260924 16:11:03.410521 23529 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 16:11:03.415161 23544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:03.417083 23529 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:11:03.417331 23529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "e06bbf5ff8244f968785bf35811e7a66"
format_stamp: "Formatted at 2026-09-24 16:11:03 on dist-test-slave-1gc3"
server_key: "9ab6e1bd1407d71460abb8d8fd9e9cd5"
server_key_iv: "ab973c7f507d0e94e12524a116eab47e"
server_key_version: "encryptionkey@0"
I20260924 16:11:03.417575 23529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:03.438360 23529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:03.443578 23529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:03.443902 23529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:03.465509 23529 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:41577
I20260924 16:11:03.465592 23595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:41577 every 8 connection(s)
I20260924 16:11:03.467065 23529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:03.472076 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 23529
I20260924 16:11:03.472339 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:11:03.472610 23596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:03.472988 3931 external_mini_cluster.cc:1428] Setting key b09ccb973e2dfd3e4a8192f2d7b4b6ff
I20260924 16:11:03.490549 23596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66: Bootstrap starting.
Sep 24 16:11:03 dist-test-slave-1gc3 krb5kdc[23516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:11:03.495249 23596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:03.496997 23596 log.cc:824] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:03.501797 23596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66: No bootstrap required, opened a new log
I20260924 16:11:03.505294 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:03.475897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39270 (local address 127.3.214.254:41577)
0924 16:11:03.477483 (+ 1586us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:03.477506 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:03.478001 (+ 495us) server_negotiation.cc:407] Connection header received
0924 16:11:03.480741 (+ 2740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:03.480786 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:03.481626 (+ 840us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:03.482321 (+ 695us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:03.483916 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:03.486044 (+ 2128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:03.487733 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:03.489012 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:03.489269 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:03.494319 (+ 5050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:03.494334 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:03.494361 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:03.494430 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:03.498793 (+ 4363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:03.499817 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:03.499830 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:03.499840 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:03.500127 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:03.500693 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:03.500699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:03.500702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:03.501631 (+ 929us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:03.502018 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:03.502834 (+ 816us) server_negotiation.cc:299] Negotiation successful
0924 16:11:03.503453 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1027,"thread_start_us":619,"threads_started":1}
I20260924 16:11:03.512058 23596 raft_consensus.cc:348] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06bbf5ff8244f968785bf35811e7a66" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 41577 } }
I20260924 16:11:03.512564 23596 raft_consensus.cc:374] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:03.512650 23596 raft_consensus.cc:729] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e06bbf5ff8244f968785bf35811e7a66, State: Initialized, Role: FOLLOWER
I20260924 16:11:03.513300 23596 consensus_queue.cc:260] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06bbf5ff8244f968785bf35811e7a66" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 41577 } }
I20260924 16:11:03.513556 23596 raft_consensus.cc:388] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:03.513670 23596 raft_consensus.cc:482] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:03.513840 23596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:03.516394 23596 raft_consensus.cc:504] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06bbf5ff8244f968785bf35811e7a66" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 41577 } }
I20260924 16:11:03.517000 23596 leader_election.cc:302] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e06bbf5ff8244f968785bf35811e7a66; no voters:
I20260924 16:11:03.517784 23596 leader_election.cc:288] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:03.518013 23601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:03.519093 23601 raft_consensus.cc:686] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66 [term 1 LEADER]: Becoming Leader. State: Replica: e06bbf5ff8244f968785bf35811e7a66, State: Running, Role: LEADER
I20260924 16:11:03.519785 23601 consensus_queue.cc:237] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06bbf5ff8244f968785bf35811e7a66" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 41577 } }
I20260924 16:11:03.520743 23596 sys_catalog.cc:564] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:03.523337 23602 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e06bbf5ff8244f968785bf35811e7a66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06bbf5ff8244f968785bf35811e7a66" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 41577 } } }
I20260924 16:11:03.523916 23602 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:03.523759 23603 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e06bbf5ff8244f968785bf35811e7a66. Latest consensus state: current_term: 1 leader_uuid: "e06bbf5ff8244f968785bf35811e7a66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06bbf5ff8244f968785bf35811e7a66" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 41577 } } }
I20260924 16:11:03.524055 23603 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:03.525090 23609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:03.532011 23609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:03.544854 23609 catalog_manager.cc:1347] Generated new cluster ID: d15ba388d51f4eb0a44959f0b84e4e11
I20260924 16:11:03.545015 23609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:03.555322 23609 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:03.557183 23609 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:03.567874 23609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66: Generated new TSK 0
I20260924 16:11:03.569152 23609 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.214.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
Entry for principal kudu/127.3.214.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab.
I20260924 16:11:03.655481 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:41577
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:04.100775 23622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:04.101225 23622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:04.101359 23622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:04.110165 23622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:04.110361 23622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:04.110500 23622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:11:04.122391 23622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.193.keytab
--principal=kudu/127.3.214.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:41577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:04.124248 23622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:04.125903 23622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:04.142836 23627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:04.142776 23630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:04.142719 23628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:04.143889 23622 server_base.cc:1034] running on GCE node
I20260924 16:11:04.144974 23622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:04.146509 23622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:04.147857 23622 hybrid_clock.cc:648] HybridClock initialized: now 1790266264147781 us; error 59 us; skew 500 ppm
I20260924 16:11:04.148339 23622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:04 dist-test-slave-1gc3 krb5kdc[23516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266264, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:04.152199 23622 init.cc:377] Logged in from keytab as kudu/127.3.214.193@KRBTEST.COM (short username kudu)
I20260924 16:11:04.154165 23622 webserver.cc:460] Webserver started at http://127.3.214.193:35671/ using document root <none> and password file <none>
I20260924 16:11:04.155035 23622 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:04.155169 23622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:04.155520 23622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:04.158828 23622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "23155d53be3d4c778a602344bceceb19"
format_stamp: "Formatted at 2026-09-24 16:11:04 on dist-test-slave-1gc3"
server_key: "71334f3ca8447b33d008f38a9793cff8"
server_key_iv: "e463130a06f80d38c57204565d70a67f"
server_key_version: "encryptionkey@0"
I20260924 16:11:04.159698 23622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "23155d53be3d4c778a602344bceceb19"
format_stamp: "Formatted at 2026-09-24 16:11:04 on dist-test-slave-1gc3"
server_key: "71334f3ca8447b33d008f38a9793cff8"
server_key_iv: "e463130a06f80d38c57204565d70a67f"
server_key_version: "encryptionkey@0"
I20260924 16:11:04.166250 23622 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260924 16:11:04.170765 23637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:04.172703 23622 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:11:04.172937 23622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "23155d53be3d4c778a602344bceceb19"
format_stamp: "Formatted at 2026-09-24 16:11:04 on dist-test-slave-1gc3"
server_key: "71334f3ca8447b33d008f38a9793cff8"
server_key_iv: "e463130a06f80d38c57204565d70a67f"
server_key_version: "encryptionkey@0"
I20260924 16:11:04.173169 23622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:04.189258 23622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:04.195149 23622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:04.195493 23622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:04.196657 23622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:11:04.198321 23622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:04.198421 23622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:04.198539 23622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:04.198612 23622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:04.238888 23622 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:41397
I20260924 16:11:04.238950 23749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:41397 every 8 connection(s)
I20260924 16:11:04.240520 23622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:11:04.247134 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 23622
I20260924 16:11:04.247386 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:11:04.247961 3931 external_mini_cluster.cc:1428] Setting key 5b196516826e5119fa22d9a0bdb9e5d2
Sep 24 16:11:04 dist-test-slave-1gc3 krb5kdc[23516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266264, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.193@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:11:04.269724 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:04.244517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:55747 (local address 127.3.214.254:41577)
0924 16:11:04.245155 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:04.245162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:04.246736 (+ 1574us) server_negotiation.cc:407] Connection header received
0924 16:11:04.249303 (+ 2567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:04.249310 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:04.249438 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:04.249556 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:04.251905 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:04.252743 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:04.254929 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:04.255446 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:04.255584 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:04.260580 (+ 4996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:04.260587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:04.260591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:04.260637 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:04.264741 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:04.266065 (+ 1324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:04.266070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:04.266073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:04.266312 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:04.266883 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:04.266888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:04.266890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:04.267244 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:04.267422 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:04.268945 (+ 1523us) server_negotiation.cc:299] Negotiation successful
0924 16:11:04.269295 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":295,"threads_started":1}
I20260924 16:11:04.270798 23753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:04.245254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:41577 (local address 127.3.214.193:55747)
0924 16:11:04.246462 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:04.246539 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:04.248893 (+ 2354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:04.249952 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:04.249967 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:04.250775 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:04.251643 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:04.251670 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:04.252973 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:04.252978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:04.254686 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:04.254712 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:04.255771 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:04.255780 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:04.256092 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:04.256952 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:04.256982 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:04.260182 (+ 3200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:04.265076 (+ 4894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:04.265090 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:04.265115 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:04.265869 (+ 754us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:04.266455 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:04.266459 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:04.266461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:04.266729 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:04.267405 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:04.267415 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:04.268266 (+ 851us) client_negotiation.cc:769] Sending connection context
0924 16:11:04.268675 (+ 409us) client_negotiation.cc:240] Negotiation successful
0924 16:11:04.269402 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":402,"threads_started":1}
I20260924 16:11:04.272907 23750 heartbeater.cc:347] Connected to a master server at 127.3.214.254:41577
I20260924 16:11:04.273496 23750 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:04.274856 23750 heartbeater.cc:511] Master 127.3.214.254:41577 requested a full tablet report, sending...
I20260924 16:11:04.278450 23561 ts_manager.cc:194] Registered new tserver with Master: 23155d53be3d4c778a602344bceceb19 (127.3.214.193:41397)
I20260924 16:11:04.280310 23561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.193@KRBTEST.COM'} at 127.3.214.193:55747
WARNING: no policy specified for kudu/127.3.214.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
Entry for principal kudu/127.3.214.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab.
I20260924 16:11:04.327855 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:41577
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:04.778272 23756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:04.778671 23756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:04.778824 23756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:04.787128 23756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:04.787277 23756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:04.787444 23756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:11:04.799435 23756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.194.keytab
--principal=kudu/127.3.214.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:41577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:04.801275 23756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:04.803105 23756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:04.820166 23761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:04.820155 23762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:04.820381 23756 server_base.cc:1034] running on GCE node
W20260924 16:11:04.820189 23764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:04.821727 23756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:04.822964 23756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:04.824265 23756 hybrid_clock.cc:648] HybridClock initialized: now 1790266264824199 us; error 51 us; skew 500 ppm
I20260924 16:11:04.824697 23756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:04 dist-test-slave-1gc3 krb5kdc[23516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266264, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:04.828740 23756 init.cc:377] Logged in from keytab as kudu/127.3.214.194@KRBTEST.COM (short username kudu)
I20260924 16:11:04.830689 23756 webserver.cc:460] Webserver started at http://127.3.214.194:34765/ using document root <none> and password file <none>
I20260924 16:11:04.831593 23756 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:04.831722 23756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:04.832108 23756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:04.835826 23756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "9c6a400eeb434817ae3d181e33af7c8a"
format_stamp: "Formatted at 2026-09-24 16:11:04 on dist-test-slave-1gc3"
server_key: "b3b1ac0a8dcbef312a698f873b9f268a"
server_key_iv: "56d3d1a37aedd281c0d42fcdaa4421b9"
server_key_version: "encryptionkey@0"
I20260924 16:11:04.836802 23756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "9c6a400eeb434817ae3d181e33af7c8a"
format_stamp: "Formatted at 2026-09-24 16:11:04 on dist-test-slave-1gc3"
server_key: "b3b1ac0a8dcbef312a698f873b9f268a"
server_key_iv: "56d3d1a37aedd281c0d42fcdaa4421b9"
server_key_version: "encryptionkey@0"
I20260924 16:11:04.843961 23756 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260924 16:11:04.849041 23771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:04.851315 23756 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20260924 16:11:04.851599 23756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "9c6a400eeb434817ae3d181e33af7c8a"
format_stamp: "Formatted at 2026-09-24 16:11:04 on dist-test-slave-1gc3"
server_key: "b3b1ac0a8dcbef312a698f873b9f268a"
server_key_iv: "56d3d1a37aedd281c0d42fcdaa4421b9"
server_key_version: "encryptionkey@0"
I20260924 16:11:04.851859 23756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:04.866953 23756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:04.871977 23756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:04.872344 23756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:04.873709 23756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:11:04.875545 23756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:04.875634 23756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:04.875766 23756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:04.875829 23756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:04.913862 23756 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:39565
I20260924 16:11:04.913893 23883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:39565 every 8 connection(s)
I20260924 16:11:04.915570 23756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:11:04.919694 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 23756
I20260924 16:11:04.919914 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:11:04.920439 3931 external_mini_cluster.cc:1428] Setting key 999b8620a7e1c51b0043a5ad11b50ca0
Sep 24 16:11:04 dist-test-slave-1gc3 krb5kdc[23516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266264, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.194@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:11:04.941475 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:04.918782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:36079 (local address 127.3.214.254:41577)
0924 16:11:04.919448 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:04.919457 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:04.920537 (+ 1080us) server_negotiation.cc:407] Connection header received
0924 16:11:04.923313 (+ 2776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:04.923318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:04.923411 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:04.923508 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:04.925241 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:04.926033 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:04.927927 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:04.928657 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:04.928882 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:04.933092 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:04.933100 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:04.933106 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:04.933177 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:04.936817 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:04.938177 (+ 1360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:04.938185 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:04.938189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:04.938415 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:04.939117 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:04.939122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:04.939125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:04.939538 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:04.939707 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:04.940853 (+ 1146us) server_negotiation.cc:299] Negotiation successful
0924 16:11:04.941125 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":282,"threads_started":1}
I20260924 16:11:04.943145 23887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:04.919362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:41577 (local address 127.3.214.194:36079)
0924 16:11:04.920295 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:04.920356 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:04.922955 (+ 2599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:04.923794 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:04.923804 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:04.924398 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:04.925051 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:04.925072 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:04.926210 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:04.926216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:04.927696 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:04.927737 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:04.929031 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:04.929038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:04.929313 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:04.930185 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:04.930213 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:04.932686 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:04.937088 (+ 4402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:04.937102 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:04.937126 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:04.937865 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:04.938565 (+ 700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:04.938570 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:04.938574 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:04.938896 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:04.939706 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:04.939716 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:04.940297 (+ 581us) client_negotiation.cc:769] Sending connection context
0924 16:11:04.940721 (+ 424us) client_negotiation.cc:240] Negotiation successful
0924 16:11:04.941375 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":368,"threads_started":1}
I20260924 16:11:04.945372 23884 heartbeater.cc:347] Connected to a master server at 127.3.214.254:41577
I20260924 16:11:04.945907 23884 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:04.947043 23884 heartbeater.cc:511] Master 127.3.214.254:41577 requested a full tablet report, sending...
I20260924 16:11:04.949256 23561 ts_manager.cc:194] Registered new tserver with Master: 9c6a400eeb434817ae3d181e33af7c8a (127.3.214.194:39565)
I20260924 16:11:04.950359 23561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.194@KRBTEST.COM'} at 127.3.214.194:36079
WARNING: no policy specified for kudu/127.3.214.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
Entry for principal kudu/127.3.214.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab.
I20260924 16:11:04.994809 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:41577
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:11:05.284791 23750 heartbeater.cc:503] Master 127.3.214.254:41577 was elected leader, sending a full tablet report...
W20260924 16:11:05.410853 23890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:05.411309 23890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:05.411473 23890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:05.419719 23890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:05.419868 23890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:05.420030 23890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:11:05.431617 23890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.195.keytab
--principal=kudu/127.3.214.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:41577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:05.433490 23890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:05.435232 23890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:05.450788 23895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:05.450868 23896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:05.451152 23898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:05.451951 23890 server_base.cc:1034] running on GCE node
I20260924 16:11:05.452807 23890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:05.454237 23890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:05.455518 23890 hybrid_clock.cc:648] HybridClock initialized: now 1790266265455449 us; error 51 us; skew 500 ppm
I20260924 16:11:05.455976 23890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:05 dist-test-slave-1gc3 krb5kdc[23516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266265, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:05.460109 23890 init.cc:377] Logged in from keytab as kudu/127.3.214.195@KRBTEST.COM (short username kudu)
I20260924 16:11:05.462086 23890 webserver.cc:460] Webserver started at http://127.3.214.195:46791/ using document root <none> and password file <none>
I20260924 16:11:05.463034 23890 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:05.463179 23890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:05.463549 23890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:05.467437 23890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "b5e86d1f4a504916b2004b80cb985ded"
format_stamp: "Formatted at 2026-09-24 16:11:05 on dist-test-slave-1gc3"
server_key: "4b25bc6e913becb386dc92138167ac95"
server_key_iv: "d2571357b76d59cfc6083fad0aab2541"
server_key_version: "encryptionkey@0"
I20260924 16:11:05.468464 23890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "b5e86d1f4a504916b2004b80cb985ded"
format_stamp: "Formatted at 2026-09-24 16:11:05 on dist-test-slave-1gc3"
server_key: "4b25bc6e913becb386dc92138167ac95"
server_key_iv: "d2571357b76d59cfc6083fad0aab2541"
server_key_version: "encryptionkey@0"
I20260924 16:11:05.475224 23890 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 16:11:05.480166 23905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:05.482291 23890 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260924 16:11:05.482525 23890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "b5e86d1f4a504916b2004b80cb985ded"
format_stamp: "Formatted at 2026-09-24 16:11:05 on dist-test-slave-1gc3"
server_key: "4b25bc6e913becb386dc92138167ac95"
server_key_iv: "d2571357b76d59cfc6083fad0aab2541"
server_key_version: "encryptionkey@0"
I20260924 16:11:05.482795 23890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:05.509136 23890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:05.514408 23890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:05.514755 23890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:05.515973 23890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:11:05.517617 23890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:05.517711 23890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:05.517845 23890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:05.517900 23890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:05.556546 23890 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:44605
I20260924 16:11:05.556658 24017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:44605 every 8 connection(s)
I20260924 16:11:05.558439 23890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:11:05.560343 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 23890
I20260924 16:11:05.560546 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:11:05.561192 3931 external_mini_cluster.cc:1428] Setting key 610f9644bb11c699acf6b839ab4d86bf
Sep 24 16:11:05 dist-test-slave-1gc3 krb5kdc[23516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266265, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.195@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:11:05.595919 24020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:05.562426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:36763 (local address 127.3.214.254:41577)
0924 16:11:05.568555 (+ 6129us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:05.568563 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:05.569912 (+ 1349us) server_negotiation.cc:407] Connection header received
0924 16:11:05.574621 (+ 4709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:05.574629 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:05.574766 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:05.574945 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:05.577170 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.578543 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.581611 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.582095 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.582269 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:05.587697 (+ 5428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:05.587705 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:05.587709 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:05.588115 (+ 406us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:05.591344 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:05.592766 (+ 1422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:05.592771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:05.592774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:05.592976 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:05.593570 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:05.593573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:05.593576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:05.593902 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:05.594039 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:05.595381 (+ 1342us) server_negotiation.cc:299] Negotiation successful
0924 16:11:05.595619 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5965,"thread_start_us":5807,"threads_started":1}
I20260924 16:11:05.598060 24021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:05.562725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:41577 (local address 127.3.214.195:36763)
0924 16:11:05.569705 (+ 6980us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:05.569782 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:05.574062 (+ 4280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:05.575287 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:05.575297 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:05.576038 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:05.576765 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:05.576787 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.578890 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.578898 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:05.581370 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:05.581403 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.582443 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.582453 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:05.582867 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:05.583800 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:05.583838 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:05.587354 (+ 3516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:05.591601 (+ 4247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:05.591618 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:05.591649 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:05.592608 (+ 959us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:05.593108 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:05.593112 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:05.593116 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:05.593438 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:05.594073 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:05.594086 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:05.594814 (+ 728us) client_negotiation.cc:769] Sending connection context
0924 16:11:05.595272 (+ 458us) client_negotiation.cc:240] Negotiation successful
0924 16:11:05.596181 (+ 909us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6517,"thread_start_us":6209,"threads_started":1}
I20260924 16:11:05.599923 24018 heartbeater.cc:347] Connected to a master server at 127.3.214.254:41577
I20260924 16:11:05.600297 24018 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:05.601145 24018 heartbeater.cc:511] Master 127.3.214.254:41577 requested a full tablet report, sending...
I20260924 16:11:05.602852 23561 ts_manager.cc:194] Registered new tserver with Master: b5e86d1f4a504916b2004b80cb985ded (127.3.214.195:44605)
I20260924 16:11:05.603857 23561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.214.195@KRBTEST.COM'} at 127.3.214.195:36763
I20260924 16:11:05.606094 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:11:05.630918 24020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:05.613576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.3.214.254:41577)
0924 16:11:05.613861 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:05.613870 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:05.614432 (+ 562us) server_negotiation.cc:407] Connection header received
0924 16:11:05.614621 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:05.614628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:05.614767 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:05.614928 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:05.616370 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.617673 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.619176 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.619871 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.620027 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:05.622951 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:05.622959 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:05.622964 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:05.623024 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:05.627047 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:05.628150 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:05.628157 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:05.628162 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:05.628395 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:05.628877 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:05.628883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:05.628886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:05.629503 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:05.629807 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:05.630191 (+ 384us) server_negotiation.cc:299] Negotiation successful
0924 16:11:05.630518 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 16:11:05.636736 23561 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:11:05.644383 23561 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:11:05.685699 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:05.661299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:39565 (local address 127.0.0.1:46182)
0924 16:11:05.664001 (+ 2702us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:05.664020 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:05.664151 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:05.677126 (+ 12975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:05.677136 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:05.677278 (+ 142us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:05.677841 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:05.677856 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.680242 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.680248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:05.682487 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:05.682503 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.684638 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.684648 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:05.684861 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:05.684923 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 16:11:05.685087 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 16:11:05.685319 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2558,"thread_start_us":2151,"threads_started":1}
I20260924 16:11:05.687281 24033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:05.663106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.3.214.194:39565)
0924 16:11:05.676566 (+ 13460us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:05.676572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:05.676595 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:11:05.676682 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:05.676688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:05.676846 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:05.677044 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:05.678063 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.680058 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.682732 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.684399 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.686340 (+ 1941us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:05.686625 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:05.686713 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 16:11:05.686902 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13284,"thread_start_us":13089,"threads_started":1}
I20260924 16:11:05.690150 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:05.661223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:41397 (local address 127.0.0.1:52880)
0924 16:11:05.662886 (+ 1663us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:05.662958 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:05.663233 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:05.678991 (+ 15758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:05.678996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:05.679022 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:05.679387 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:05.679399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.683216 (+ 3817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.683222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:05.685153 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:05.685167 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.689231 (+ 4064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.689239 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:05.689427 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:05.689467 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 16:11:05.689559 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 16:11:05.689729 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1264,"mutex_wait_us":4,"thread_start_us":2429,"threads_started":1}
I20260924 16:11:05.692150 23819 tablet_service.cc:1467] Processing CreateTablet for tablet 118e64682f5044979f0589cb04b6cd18 (DEFAULT_TABLE table=test-table [id=0603345a1a224cde86b589eab8e74126]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:05.694291 23819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 118e64682f5044979f0589cb04b6cd18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:05.694941 24034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:05.661905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52880 (local address 127.3.214.193:41397)
0924 16:11:05.675185 (+ 13280us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:05.675197 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:05.675250 (+ 53us) server_negotiation.cc:407] Connection header received
0924 16:11:05.675409 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:05.675419 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:05.675779 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:05.679623 (+ 3844us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:05.679875 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.683052 (+ 3177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.685367 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.688961 (+ 3594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.692661 (+ 3700us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:05.692967 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:05.694124 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0924 16:11:05.694459 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13066,"thread_start_us":12886,"threads_started":1}
I20260924 16:11:05.700457 23685 tablet_service.cc:1467] Processing CreateTablet for tablet 118e64682f5044979f0589cb04b6cd18 (DEFAULT_TABLE table=test-table [id=0603345a1a224cde86b589eab8e74126]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:05.702793 23685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 118e64682f5044979f0589cb04b6cd18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:05.704589 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:05.665342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:44605 (local address 127.0.0.1:41304)
0924 16:11:05.678635 (+ 13293us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:05.678707 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:05.678833 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:05.694863 (+ 16030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:05.694868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:05.694903 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:05.695507 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:05.695521 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.699363 (+ 3842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.699369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:05.701559 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:05.701580 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.703719 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.703726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:05.703922 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:05.703962 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 16:11:05.704035 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 16:11:05.704190 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13140,"thread_start_us":12253,"threads_started":1}
I20260924 16:11:05.707882 24036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:05.665866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41304 (local address 127.3.214.195:44605)
0924 16:11:05.693922 (+ 28056us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:05.693932 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:05.693974 (+ 42us) server_negotiation.cc:407] Connection header received
0924 16:11:05.694136 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:05.694145 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:05.694590 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:05.695718 (+ 1128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:05.696050 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.699180 (+ 3130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.701814 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.703465 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.706817 (+ 3352us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:05.707129 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:05.707217 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 16:11:05.707507 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27818,"thread_start_us":27630,"threads_started":1}
I20260924 16:11:05.713819 23953 tablet_service.cc:1467] Processing CreateTablet for tablet 118e64682f5044979f0589cb04b6cd18 (DEFAULT_TABLE table=test-table [id=0603345a1a224cde86b589eab8e74126]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:05.718981 23953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 118e64682f5044979f0589cb04b6cd18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:05.723306 24037 tablet_bootstrap.cc:492] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a: Bootstrap starting.
I20260924 16:11:05.726935 24038 tablet_bootstrap.cc:492] T 118e64682f5044979f0589cb04b6cd18 P 23155d53be3d4c778a602344bceceb19: Bootstrap starting.
I20260924 16:11:05.728520 24037 tablet_bootstrap.cc:654] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:05.730405 24037 log.cc:824] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:05.731346 24038 tablet_bootstrap.cc:654] T 118e64682f5044979f0589cb04b6cd18 P 23155d53be3d4c778a602344bceceb19: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:05.732954 24038 log.cc:824] T 118e64682f5044979f0589cb04b6cd18 P 23155d53be3d4c778a602344bceceb19: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:05.741438 24038 tablet_bootstrap.cc:492] T 118e64682f5044979f0589cb04b6cd18 P 23155d53be3d4c778a602344bceceb19: No bootstrap required, opened a new log
I20260924 16:11:05.741526 24037 tablet_bootstrap.cc:492] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a: No bootstrap required, opened a new log
I20260924 16:11:05.741845 24038 ts_tablet_manager.cc:1397] T 118e64682f5044979f0589cb04b6cd18 P 23155d53be3d4c778a602344bceceb19: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260924 16:11:05.741964 24037 ts_tablet_manager.cc:1397] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.005s
I20260924 16:11:05.755033 24041 tablet_bootstrap.cc:492] T 118e64682f5044979f0589cb04b6cd18 P b5e86d1f4a504916b2004b80cb985ded: Bootstrap starting.
I20260924 16:11:05.755036 24037 raft_consensus.cc:348] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23155d53be3d4c778a602344bceceb19" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 41397 } } peers { permanent_uuid: "9c6a400eeb434817ae3d181e33af7c8a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 39565 } } peers { permanent_uuid: "b5e86d1f4a504916b2004b80cb985ded" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 44605 } }
I20260924 16:11:05.755692 24037 raft_consensus.cc:374] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:05.755797 24037 raft_consensus.cc:729] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6a400eeb434817ae3d181e33af7c8a, State: Initialized, Role: FOLLOWER
I20260924 16:11:05.756785 24037 consensus_queue.cc:260] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23155d53be3d4c778a602344bceceb19" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 41397 } } peers { permanent_uuid: "9c6a400eeb434817ae3d181e33af7c8a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 39565 } } peers { permanent_uuid: "b5e86d1f4a504916b2004b80cb985ded" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 44605 } }
I20260924 16:11:05.758178 24038 raft_consensus.cc:348] T 118e64682f5044979f0589cb04b6cd18 P 23155d53be3d4c778a602344bceceb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23155d53be3d4c778a602344bceceb19" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 41397 } } peers { permanent_uuid: "9c6a400eeb434817ae3d181e33af7c8a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 39565 } } peers { permanent_uuid: "b5e86d1f4a504916b2004b80cb985ded" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 44605 } }
I20260924 16:11:05.758767 24038 raft_consensus.cc:374] T 118e64682f5044979f0589cb04b6cd18 P 23155d53be3d4c778a602344bceceb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:05.758877 24038 raft_consensus.cc:729] T 118e64682f5044979f0589cb04b6cd18 P 23155d53be3d4c778a602344bceceb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23155d53be3d4c778a602344bceceb19, State: Initialized, Role: FOLLOWER
I20260924 16:11:05.759670 24038 consensus_queue.cc:260] T 118e64682f5044979f0589cb04b6cd18 P 23155d53be3d4c778a602344bceceb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23155d53be3d4c778a602344bceceb19" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 41397 } } peers { permanent_uuid: "9c6a400eeb434817ae3d181e33af7c8a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 39565 } } peers { permanent_uuid: "b5e86d1f4a504916b2004b80cb985ded" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 44605 } }
I20260924 16:11:05.760969 24041 tablet_bootstrap.cc:654] T 118e64682f5044979f0589cb04b6cd18 P b5e86d1f4a504916b2004b80cb985ded: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:05.762621 24041 log.cc:824] T 118e64682f5044979f0589cb04b6cd18 P b5e86d1f4a504916b2004b80cb985ded: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:05.765944 24037 ts_tablet_manager.cc:1428] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a: Time spent starting tablet: real 0.024s user 0.021s sys 0.004s
I20260924 16:11:05.766067 23884 heartbeater.cc:503] Master 127.3.214.254:41577 was elected leader, sending a full tablet report...
I20260924 16:11:05.770685 24038 ts_tablet_manager.cc:1428] T 118e64682f5044979f0589cb04b6cd18 P 23155d53be3d4c778a602344bceceb19: Time spent starting tablet: real 0.029s user 0.021s sys 0.005s
I20260924 16:11:05.774840 24041 tablet_bootstrap.cc:492] T 118e64682f5044979f0589cb04b6cd18 P b5e86d1f4a504916b2004b80cb985ded: No bootstrap required, opened a new log
I20260924 16:11:05.775198 24041 ts_tablet_manager.cc:1397] T 118e64682f5044979f0589cb04b6cd18 P b5e86d1f4a504916b2004b80cb985ded: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.010s
I20260924 16:11:05.783581 24041 raft_consensus.cc:348] T 118e64682f5044979f0589cb04b6cd18 P b5e86d1f4a504916b2004b80cb985ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23155d53be3d4c778a602344bceceb19" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 41397 } } peers { permanent_uuid: "9c6a400eeb434817ae3d181e33af7c8a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 39565 } } peers { permanent_uuid: "b5e86d1f4a504916b2004b80cb985ded" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 44605 } }
I20260924 16:11:05.784060 24041 raft_consensus.cc:374] T 118e64682f5044979f0589cb04b6cd18 P b5e86d1f4a504916b2004b80cb985ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:05.784162 24041 raft_consensus.cc:729] T 118e64682f5044979f0589cb04b6cd18 P b5e86d1f4a504916b2004b80cb985ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5e86d1f4a504916b2004b80cb985ded, State: Initialized, Role: FOLLOWER
I20260924 16:11:05.784827 24041 consensus_queue.cc:260] T 118e64682f5044979f0589cb04b6cd18 P b5e86d1f4a504916b2004b80cb985ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23155d53be3d4c778a602344bceceb19" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 41397 } } peers { permanent_uuid: "9c6a400eeb434817ae3d181e33af7c8a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 39565 } } peers { permanent_uuid: "b5e86d1f4a504916b2004b80cb985ded" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 44605 } }
I20260924 16:11:05.786335 24018 heartbeater.cc:503] Master 127.3.214.254:41577 was elected leader, sending a full tablet report...
I20260924 16:11:05.786844 24041 ts_tablet_manager.cc:1428] T 118e64682f5044979f0589cb04b6cd18 P b5e86d1f4a504916b2004b80cb985ded: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
W20260924 16:11:05.810742 24019 tablet.cc:2369] T 118e64682f5044979f0589cb04b6cd18 P b5e86d1f4a504916b2004b80cb985ded: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:11:05.890523 24042 raft_consensus.cc:482] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:11:05.890887 24042 raft_consensus.cc:504] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23155d53be3d4c778a602344bceceb19" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 41397 } } peers { permanent_uuid: "9c6a400eeb434817ae3d181e33af7c8a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 39565 } } peers { permanent_uuid: "b5e86d1f4a504916b2004b80cb985ded" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 44605 } }
I20260924 16:11:05.892876 24042 leader_election.cc:288] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23155d53be3d4c778a602344bceceb19 (127.3.214.193:41397), b5e86d1f4a504916b2004b80cb985ded (127.3.214.195:44605)
I20260924 16:11:05.900532 24036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:05.893421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:49139 (local address 127.3.214.195:44605)
0924 16:11:05.893635 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:05.893642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:05.894772 (+ 1130us) server_negotiation.cc:407] Connection header received
0924 16:11:05.894981 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:05.894987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:05.895101 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:05.895215 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:05.895948 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.896766 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.898701 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.899489 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.899660 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:05.899743 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:05.900108 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 16:11:05.900260 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 16:11:05.900601 24034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:05.893709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:34101 (local address 127.3.214.193:41397)
0924 16:11:05.893972 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:05.893979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:05.894473 (+ 494us) server_negotiation.cc:407] Connection header received
0924 16:11:05.894756 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:05.894763 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:05.894851 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:05.894949 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:05.895623 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.896513 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.898618 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.899478 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.899662 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:05.899751 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:05.900190 (+ 439us) server_negotiation.cc:299] Negotiation successful
0924 16:11:05.900338 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 16:11:05.900888 24047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:05.893582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:44605 (local address 127.3.214.194:49139)
0924 16:11:05.894617 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:05.894636 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:05.894829 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:05.895258 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:05.895261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:05.895283 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:05.895782 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:05.895795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.896921 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.896925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:05.898532 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:05.898548 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.899750 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.899757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:05.899912 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:05.899961 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 16:11:05.900043 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 16:11:05.900180 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":1,"thread_start_us":601,"threads_started":1}
I20260924 16:11:05.900966 24046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:05.893347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:41397 (local address 127.3.214.194:34101)
0924 16:11:05.894311 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:05.894339 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:05.894559 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:05.894997 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:05.895002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:05.895033 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:05.895447 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:05.895460 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.896674 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.896680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:05.898432 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:05.898448 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.899804 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.899810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:05.899990 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:05.900023 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 16:11:05.900133 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 16:11:05.900285 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":29,"thread_start_us":472,"threads_started":1}
I20260924 16:11:05.902271 23973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118e64682f5044979f0589cb04b6cd18" candidate_uuid: "9c6a400eeb434817ae3d181e33af7c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e86d1f4a504916b2004b80cb985ded" is_pre_election: true
I20260924 16:11:05.902271 23705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118e64682f5044979f0589cb04b6cd18" candidate_uuid: "9c6a400eeb434817ae3d181e33af7c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23155d53be3d4c778a602344bceceb19" is_pre_election: true
I20260924 16:11:05.903003 23705 raft_consensus.cc:2454] T 118e64682f5044979f0589cb04b6cd18 P 23155d53be3d4c778a602344bceceb19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c6a400eeb434817ae3d181e33af7c8a in term 0.
I20260924 16:11:05.903002 23973 raft_consensus.cc:2454] T 118e64682f5044979f0589cb04b6cd18 P b5e86d1f4a504916b2004b80cb985ded [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c6a400eeb434817ae3d181e33af7c8a in term 0.
I20260924 16:11:05.903973 23772 leader_election.cc:302] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23155d53be3d4c778a602344bceceb19, 9c6a400eeb434817ae3d181e33af7c8a; no voters:
I20260924 16:11:05.904577 24042 raft_consensus.cc:2781] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:11:05.904799 24042 raft_consensus.cc:482] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:11:05.904899 24042 raft_consensus.cc:3037] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:05.907382 24042 raft_consensus.cc:504] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23155d53be3d4c778a602344bceceb19" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 41397 } } peers { permanent_uuid: "9c6a400eeb434817ae3d181e33af7c8a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 39565 } } peers { permanent_uuid: "b5e86d1f4a504916b2004b80cb985ded" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 44605 } }
I20260924 16:11:05.908171 24042 leader_election.cc:288] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a [CANDIDATE]: Term 1 election: Requested vote from peers 23155d53be3d4c778a602344bceceb19 (127.3.214.193:41397), b5e86d1f4a504916b2004b80cb985ded (127.3.214.195:44605)
I20260924 16:11:05.908897 23973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118e64682f5044979f0589cb04b6cd18" candidate_uuid: "9c6a400eeb434817ae3d181e33af7c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e86d1f4a504916b2004b80cb985ded"
I20260924 16:11:05.908952 23705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118e64682f5044979f0589cb04b6cd18" candidate_uuid: "9c6a400eeb434817ae3d181e33af7c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23155d53be3d4c778a602344bceceb19"
I20260924 16:11:05.909159 23973 raft_consensus.cc:3037] T 118e64682f5044979f0589cb04b6cd18 P b5e86d1f4a504916b2004b80cb985ded [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:05.909233 23705 raft_consensus.cc:3037] T 118e64682f5044979f0589cb04b6cd18 P 23155d53be3d4c778a602344bceceb19 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:05.911594 23973 raft_consensus.cc:2454] T 118e64682f5044979f0589cb04b6cd18 P b5e86d1f4a504916b2004b80cb985ded [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c6a400eeb434817ae3d181e33af7c8a in term 1.
I20260924 16:11:05.912030 23705 raft_consensus.cc:2454] T 118e64682f5044979f0589cb04b6cd18 P 23155d53be3d4c778a602344bceceb19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c6a400eeb434817ae3d181e33af7c8a in term 1.
I20260924 16:11:05.912204 23773 leader_election.cc:302] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c6a400eeb434817ae3d181e33af7c8a, b5e86d1f4a504916b2004b80cb985ded; no voters:
I20260924 16:11:05.912652 24042 raft_consensus.cc:2781] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:05.913228 24042 raft_consensus.cc:686] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a [term 1 LEADER]: Becoming Leader. State: Replica: 9c6a400eeb434817ae3d181e33af7c8a, State: Running, Role: LEADER
I20260924 16:11:05.913879 24042 consensus_queue.cc:237] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23155d53be3d4c778a602344bceceb19" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 41397 } } peers { permanent_uuid: "9c6a400eeb434817ae3d181e33af7c8a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 39565 } } peers { permanent_uuid: "b5e86d1f4a504916b2004b80cb985ded" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 44605 } }
W20260924 16:11:05.918175 23885 tablet.cc:2369] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:11:05.919138 23561 catalog_manager.cc:5495] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c6a400eeb434817ae3d181e33af7c8a (127.3.214.194). New cstate: current_term: 1 leader_uuid: "9c6a400eeb434817ae3d181e33af7c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23155d53be3d4c778a602344bceceb19" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 41397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c6a400eeb434817ae3d181e33af7c8a" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 39565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5e86d1f4a504916b2004b80cb985ded" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 44605 } health_report { overall_health: UNKNOWN } } }
I20260924 16:11:05.990310 24033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:05.984045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46190 (local address 127.3.214.194:39565)
0924 16:11:05.984330 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:05.984336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:05.984358 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:11:05.984578 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:05.984583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:05.984707 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:05.984870 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:05.985223 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.986018 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.987583 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:05.988063 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:05.988199 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:05.988898 (+ 699us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:05.989606 (+ 708us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:05.989781 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:05.989876 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 16:11:05.989977 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260924 16:11:05.994365 23751 tablet.cc:2369] T 118e64682f5044979f0589cb04b6cd18 P 23155d53be3d4c778a602344bceceb19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:11:06.001163 23705 raft_consensus.cc:1261] T 118e64682f5044979f0589cb04b6cd18 P 23155d53be3d4c778a602344bceceb19 [term 1 FOLLOWER]: Refusing update from remote peer 9c6a400eeb434817ae3d181e33af7c8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:06.001226 23973 raft_consensus.cc:1261] T 118e64682f5044979f0589cb04b6cd18 P b5e86d1f4a504916b2004b80cb985ded [term 1 FOLLOWER]: Refusing update from remote peer 9c6a400eeb434817ae3d181e33af7c8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:06.002609 24048 consensus_queue.cc:1035] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e86d1f4a504916b2004b80cb985ded" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 44605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:11:06.003116 24042 consensus_queue.cc:1035] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "23155d53be3d4c778a602344bceceb19" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 41397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:11:06.017336 24051 mvcc.cc:205] Tried to move back new op lower bound from 7332930625525784576 to 7332930625187811328. Current Snapshot: MvccSnapshot[applied={T|T < 7332930625525784576}]
I20260924 16:11:06.038276 23561 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:11:06.038606 23561 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:11:06.043570 23561 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e06bbf5ff8244f968785bf35811e7a66: Sending DeleteTablet for 3 replicas of tablet 118e64682f5044979f0589cb04b6cd18
I20260924 16:11:06.045239 23685 tablet_service.cc:1514] Processing DeleteTablet for tablet 118e64682f5044979f0589cb04b6cd18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:06 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:52880
I20260924 16:11:06.045357 23819 tablet_service.cc:1514] Processing DeleteTablet for tablet 118e64682f5044979f0589cb04b6cd18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:06 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:46182
I20260924 16:11:06.046304 23953 tablet_service.cc:1514] Processing DeleteTablet for tablet 118e64682f5044979f0589cb04b6cd18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:06 UTC) from {username='kudu', principal='kudu/127.3.214.254@KRBTEST.COM'} at 127.0.0.1:41304
I20260924 16:11:06.048465 24063 tablet_replica.cc:330] T 118e64682f5044979f0589cb04b6cd18 P b5e86d1f4a504916b2004b80cb985ded: stopping tablet replica
I20260924 16:11:06.049139 24063 raft_consensus.cc:2229] T 118e64682f5044979f0589cb04b6cd18 P b5e86d1f4a504916b2004b80cb985ded [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:11:06.049858 24063 raft_consensus.cc:2258] T 118e64682f5044979f0589cb04b6cd18 P b5e86d1f4a504916b2004b80cb985ded [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:06.050706 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 23622
I20260924 16:11:06.051169 24061 tablet_replica.cc:330] T 118e64682f5044979f0589cb04b6cd18 P 23155d53be3d4c778a602344bceceb19: stopping tablet replica
I20260924 16:11:06.051790 24061 raft_consensus.cc:2229] T 118e64682f5044979f0589cb04b6cd18 P 23155d53be3d4c778a602344bceceb19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:11:06.051841 24062 tablet_replica.cc:330] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a: stopping tablet replica
I20260924 16:11:06.052470 24061 raft_consensus.cc:2258] T 118e64682f5044979f0589cb04b6cd18 P 23155d53be3d4c778a602344bceceb19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:06.052665 24062 raft_consensus.cc:2229] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:11:06.056115 24063 ts_tablet_manager.cc:1905] T 118e64682f5044979f0589cb04b6cd18 P b5e86d1f4a504916b2004b80cb985ded: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:06.056663 24061 ts_tablet_manager.cc:1905] T 118e64682f5044979f0589cb04b6cd18 P 23155d53be3d4c778a602344bceceb19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:06.062693 24062 raft_consensus.cc:2258] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:06.066222 24062 ts_tablet_manager.cc:1905] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:06.066290 24061 ts_tablet_manager.cc:1918] T 118e64682f5044979f0589cb04b6cd18 P 23155d53be3d4c778a602344bceceb19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:11:06.066541 24061 log.cc:1196] T 118e64682f5044979f0589cb04b6cd18 P 23155d53be3d4c778a602344bceceb19: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-0/wal/wals/118e64682f5044979f0589cb04b6cd18
I20260924 16:11:06.066727 24063 ts_tablet_manager.cc:1918] T 118e64682f5044979f0589cb04b6cd18 P b5e86d1f4a504916b2004b80cb985ded: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:11:06.066953 24063 log.cc:1196] T 118e64682f5044979f0589cb04b6cd18 P b5e86d1f4a504916b2004b80cb985ded: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/118e64682f5044979f0589cb04b6cd18
I20260924 16:11:06.067171 24061 ts_tablet_manager.cc:1939] T 118e64682f5044979f0589cb04b6cd18 P 23155d53be3d4c778a602344bceceb19: Deleting consensus metadata
I20260924 16:11:06.067484 24063 ts_tablet_manager.cc:1939] T 118e64682f5044979f0589cb04b6cd18 P b5e86d1f4a504916b2004b80cb985ded: Deleting consensus metadata
I20260924 16:11:06.069819 23546 catalog_manager.cc:4841] TS b5e86d1f4a504916b2004b80cb985ded (127.3.214.195:44605): tablet 118e64682f5044979f0589cb04b6cd18 (table test-table [id=0603345a1a224cde86b589eab8e74126]) successfully deleted
I20260924 16:11:06.070323 23545 catalog_manager.cc:4841] TS 23155d53be3d4c778a602344bceceb19 (127.3.214.193:41397): tablet 118e64682f5044979f0589cb04b6cd18 (table test-table [id=0603345a1a224cde86b589eab8e74126]) successfully deleted
I20260924 16:11:06.075378 24062 ts_tablet_manager.cc:1918] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:11:06.075613 24062 log.cc:1196] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/118e64682f5044979f0589cb04b6cd18
I20260924 16:11:06.076195 24062 ts_tablet_manager.cc:1939] T 118e64682f5044979f0589cb04b6cd18 P 9c6a400eeb434817ae3d181e33af7c8a: Deleting consensus metadata
I20260924 16:11:06.078451 23547 catalog_manager.cc:4841] TS 9c6a400eeb434817ae3d181e33af7c8a (127.3.214.194:39565): tablet 118e64682f5044979f0589cb04b6cd18 (table test-table [id=0603345a1a224cde86b589eab8e74126]) successfully deleted
Sep 24 16:11:06 dist-test-slave-1gc3 krb5kdc[23516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.193@KRBTEST.COM
I20260924 16:11:06.091297 24034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:06.053085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52882 (local address 127.3.214.193:41397)
0924 16:11:06.053305 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:06.053312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:06.063928 (+ 10616us) server_negotiation.cc:407] Connection header received
0924 16:11:06.070301 (+ 6373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:06.070308 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:06.070416 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:06.070509 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:06.072283 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:06.073101 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:06.074605 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:06.075080 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:06.075162 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:06.084216 (+ 9054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:06.084242 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:06.084269 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:06.084325 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:06.087885 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:06.088861 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:06.088871 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:06.088873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:06.089038 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:06.089445 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:06.089449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:06.089451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:06.089910 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:06.090274 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:06.090625 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 16:11:06.090890 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:11:06.092304 23745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:06.195400 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 23622
I20260924 16:11:06.222925 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 23756
Sep 24 16:11:06 dist-test-slave-1gc3 krb5kdc[23516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.194@KRBTEST.COM
I20260924 16:11:06.242494 24033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:06.224107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46202 (local address 127.3.214.194:39565)
0924 16:11:06.224367 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:06.224372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:06.224395 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:11:06.224467 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:06.224475 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:06.224581 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:06.224679 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:06.226213 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:06.227265 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:06.228765 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:06.229369 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:06.229512 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:06.235100 (+ 5588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:06.235126 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:06.235152 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:06.235219 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:06.238905 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:06.239868 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:06.239878 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:06.239880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:06.240105 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:06.240529 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:06.240533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:06.240535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:06.241010 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:06.241373 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:06.241694 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 16:11:06.241989 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 16:11:06.243536 23879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:06.379365 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 23756
I20260924 16:11:06.403110 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 23890
Sep 24 16:11:06 dist-test-slave-1gc3 krb5kdc[23516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.195@KRBTEST.COM
I20260924 16:11:06.423728 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:06.404106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41310 (local address 127.3.214.195:44605)
0924 16:11:06.404684 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:06.404689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:06.404711 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:11:06.404806 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:06.404810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:06.404901 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:06.405019 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:06.406696 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:06.407545 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:06.409544 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:06.410091 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:06.410403 (+ 312us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:06.416367 (+ 5964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:06.416396 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:06.416422 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:06.416480 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:06.419910 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:06.420983 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:06.420993 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:06.420996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:06.421222 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:06.421754 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:06.421758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:06.421760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:06.422311 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:06.422621 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:06.423008 (+ 387us) server_negotiation.cc:299] Negotiation successful
0924 16:11:06.423279 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":325,"threads_started":1}
I20260924 16:11:06.424525 24013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:06.526090 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 23890
I20260924 16:11:06.543501 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 23529
I20260924 16:11:06.544924 23591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:06.665299 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 23529
2026-09-24T16:11:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (5083 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T16:11:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:06Z Disabled control of system clock
I20260924 16:11:06.762745 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:37233
--webserver_interface=127.3.214.254
--webserver_port=0
--builtin_ntp_servers=127.3.214.212:36369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:37233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:11:07.184589 24073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:07.185005 24073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:07.185111 24073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:07.193594 24073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:07.193758 24073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:07.193800 24073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:07.193876 24073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:07.193946 24073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:07.205962 24073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:37233
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:37233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:07.207830 24073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:07.210218 24073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:07.224334 24078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:07.224642 24079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:07.224637 24081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:07.225467 24073 server_base.cc:1034] running on GCE node
I20260924 16:11:07.226593 24073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:07.228490 24073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:07.229727 24073 hybrid_clock.cc:648] HybridClock initialized: now 1790266267229666 us; error 63 us; skew 500 ppm
I20260924 16:11:07.230199 24073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:11:07.233165 24073 webserver.cc:460] Webserver started at http://127.3.214.254:32845/ using document root <none> and password file <none>
I20260924 16:11:07.234035 24073 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:07.234187 24073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:07.234562 24073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:07.238209 24073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "0d2195aa518d4de29bcdb7408a0e760b"
format_stamp: "Formatted at 2026-09-24 16:11:07 on dist-test-slave-1gc3"
server_key: "7caaa1e1630b56656b38baeab71b1f6d"
server_key_iv: "cb1063c0b71f27e415ed2645848b00b3"
server_key_version: "encryptionkey@0"
I20260924 16:11:07.239148 24073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "0d2195aa518d4de29bcdb7408a0e760b"
format_stamp: "Formatted at 2026-09-24 16:11:07 on dist-test-slave-1gc3"
server_key: "7caaa1e1630b56656b38baeab71b1f6d"
server_key_iv: "cb1063c0b71f27e415ed2645848b00b3"
server_key_version: "encryptionkey@0"
I20260924 16:11:07.245720 24073 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260924 16:11:07.250418 24087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:07.252203 24073 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:11:07.252455 24073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "0d2195aa518d4de29bcdb7408a0e760b"
format_stamp: "Formatted at 2026-09-24 16:11:07 on dist-test-slave-1gc3"
server_key: "7caaa1e1630b56656b38baeab71b1f6d"
server_key_iv: "cb1063c0b71f27e415ed2645848b00b3"
server_key_version: "encryptionkey@0"
I20260924 16:11:07.252724 24073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:07.272085 24073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:07.273178 24073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:07.273512 24073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:07.295066 24073 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:37233
I20260924 16:11:07.295153 24138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:37233 every 8 connection(s)
I20260924 16:11:07.296761 24073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:07.300611 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 24073
I20260924 16:11:07.300794 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:11:07.301319 3931 external_mini_cluster.cc:1428] Setting key 56808bcb49217c4f411290c09d313547
I20260924 16:11:07.301482 24139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:07.317617 24139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b: Bootstrap starting.
I20260924 16:11:07.318307 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:07.303861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49478 (local address 127.3.214.254:37233)
0924 16:11:07.305154 (+ 1293us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:07.305169 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:07.305237 (+ 68us) server_negotiation.cc:407] Connection header received
0924 16:11:07.307188 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:07.307219 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:07.307851 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:07.308287 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:07.309155 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:07.310700 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:07.312633 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:07.313510 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:07.313689 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:07.315234 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:07.315245 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:07.315266 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:07.315969 (+ 703us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:07.316072 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:07.316099 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:07.316204 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:07.316485 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 16:11:07.316915 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":455,"threads_started":1}
I20260924 16:11:07.321689 24139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:07.322934 24139 log.cc:824] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:07.326717 24139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b: No bootstrap required, opened a new log
I20260924 16:11:07.335343 24139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2195aa518d4de29bcdb7408a0e760b" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 37233 } }
I20260924 16:11:07.335964 24139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:07.336123 24139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d2195aa518d4de29bcdb7408a0e760b, State: Initialized, Role: FOLLOWER
I20260924 16:11:07.336930 24139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2195aa518d4de29bcdb7408a0e760b" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 37233 } }
I20260924 16:11:07.337255 24139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:07.337409 24139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:07.337592 24139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:07.340050 24139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2195aa518d4de29bcdb7408a0e760b" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 37233 } }
I20260924 16:11:07.340720 24139 leader_election.cc:302] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d2195aa518d4de29bcdb7408a0e760b; no voters:
I20260924 16:11:07.341527 24139 leader_election.cc:288] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:07.341785 24144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:07.343076 24144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b [term 1 LEADER]: Becoming Leader. State: Replica: 0d2195aa518d4de29bcdb7408a0e760b, State: Running, Role: LEADER
I20260924 16:11:07.343838 24144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2195aa518d4de29bcdb7408a0e760b" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 37233 } }
I20260924 16:11:07.344187 24139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:07.347553 24146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d2195aa518d4de29bcdb7408a0e760b. Latest consensus state: current_term: 1 leader_uuid: "0d2195aa518d4de29bcdb7408a0e760b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2195aa518d4de29bcdb7408a0e760b" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 37233 } } }
I20260924 16:11:07.348093 24146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:07.347651 24145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d2195aa518d4de29bcdb7408a0e760b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2195aa518d4de29bcdb7408a0e760b" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 37233 } } }
I20260924 16:11:07.348384 24145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:07.351691 24151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:07.358894 24151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:07.371784 24151 catalog_manager.cc:1347] Generated new cluster ID: e1181e1b7bf64aba991e937b5853291b
I20260924 16:11:07.371966 24151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:07.390738 24151 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:07.393021 24151 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:07.414904 24151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b: Generated new TSK 0
I20260924 16:11:07.416458 24151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:11:07.431627 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.193:0
--local_ip_for_outbound_sockets=127.3.214.193
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:37233
--builtin_ntp_servers=127.3.214.212:36369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:11:07.815142 24163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:07.815526 24163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:07.815665 24163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:07.823685 24163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:07.823822 24163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:07.823978 24163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.193
I20260924 16:11:07.833632 24163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.214.193
--webserver_port=0
--tserver_master_addrs=127.3.214.254:37233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.193
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:07.835114 24163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:07.837044 24163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:07.853134 24171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:07.853159 24169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:07.853178 24168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:07.854171 24163 server_base.cc:1034] running on GCE node
I20260924 16:11:07.854902 24163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:07.856038 24163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:07.857302 24163 hybrid_clock.cc:648] HybridClock initialized: now 1790266267857208 us; error 62 us; skew 500 ppm
I20260924 16:11:07.857682 24163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:11:07.860460 24163 webserver.cc:460] Webserver started at http://127.3.214.193:37099/ using document root <none> and password file <none>
I20260924 16:11:07.861233 24163 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:07.861346 24163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:07.861682 24163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:07.864845 24163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-0/data/instance:
uuid: "e3e155fda5e4469eb2a7cf4626c4e14b"
format_stamp: "Formatted at 2026-09-24 16:11:07 on dist-test-slave-1gc3"
server_key: "ab60f9e42f9aa47ba770bc86f9970073"
server_key_iv: "323d2cdbcea70ba3bcb2383ad9b38425"
server_key_version: "encryptionkey@0"
I20260924 16:11:07.865684 24163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance:
uuid: "e3e155fda5e4469eb2a7cf4626c4e14b"
format_stamp: "Formatted at 2026-09-24 16:11:07 on dist-test-slave-1gc3"
server_key: "ab60f9e42f9aa47ba770bc86f9970073"
server_key_iv: "323d2cdbcea70ba3bcb2383ad9b38425"
server_key_version: "encryptionkey@0"
I20260924 16:11:07.871903 24163 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 16:11:07.876390 24177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:07.878213 24163 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:11:07.878420 24163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-0/wal
uuid: "e3e155fda5e4469eb2a7cf4626c4e14b"
format_stamp: "Formatted at 2026-09-24 16:11:07 on dist-test-slave-1gc3"
server_key: "ab60f9e42f9aa47ba770bc86f9970073"
server_key_iv: "323d2cdbcea70ba3bcb2383ad9b38425"
server_key_version: "encryptionkey@0"
I20260924 16:11:07.878664 24163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:07.903530 24163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:07.904430 24163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:07.904747 24163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:07.905867 24163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:11:07.907589 24163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:07.907689 24163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:07.907797 24163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:07.907868 24163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:07.947893 24163 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.193:40899
I20260924 16:11:07.947966 24289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.193:40899 every 8 connection(s)
I20260924 16:11:07.949640 24163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-0/data/info.pb
I20260924 16:11:07.953178 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 24163
I20260924 16:11:07.953392 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-0/wal/instance
I20260924 16:11:07.953907 3931 external_mini_cluster.cc:1428] Setting key 814ad3ce05b08e518d5a96acd3bd2a59
I20260924 16:11:07.956746 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.194:0
--local_ip_for_outbound_sockets=127.3.214.194
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:37233
--builtin_ntp_servers=127.3.214.212:36369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:11:07.965191 24292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:07.952876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.193:46797 (local address 127.3.214.254:37233)
0924 16:11:07.953620 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:07.953638 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:07.955128 (+ 1490us) server_negotiation.cc:407] Connection header received
0924 16:11:07.957349 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:07.957356 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:07.957475 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:07.957599 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:07.959080 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:07.959953 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:07.961696 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:07.962218 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:07.962346 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:07.964142 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:07.964148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:07.964152 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:07.964184 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:07.964240 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:07.964255 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:07.964354 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:07.964627 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 16:11:07.964776 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":344,"threads_started":1}
I20260924 16:11:07.966652 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:07.953608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:37233 (local address 127.3.214.193:46797)
0924 16:11:07.954932 (+ 1324us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:07.954985 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:07.957021 (+ 2036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:07.957889 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:07.957900 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:07.958021 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:07.958837 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:07.958870 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:07.960135 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:07.960139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:07.961530 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:07.961545 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:07.962481 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:07.962489 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:07.962766 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:07.963695 (+ 929us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:11:07.963698 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:07.963759 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:11:07.963880 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:07.964377 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:07.964391 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:07.964399 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 16:11:07.964602 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 16:11:07.965190 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":400,"threads_started":1}
I20260924 16:11:07.969041 24290 heartbeater.cc:347] Connected to a master server at 127.3.214.254:37233
I20260924 16:11:07.969547 24290 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:07.970738 24290 heartbeater.cc:511] Master 127.3.214.254:37233 requested a full tablet report, sending...
I20260924 16:11:07.974089 24104 ts_manager.cc:194] Registered new tserver with Master: e3e155fda5e4469eb2a7cf4626c4e14b (127.3.214.193:40899)
I20260924 16:11:07.976292 24104 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.214.193:46797
W20260924 16:11:08.330596 24294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:08.330940 24294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:08.331041 24294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:08.340143 24294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:08.340289 24294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:08.340409 24294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.194
I20260924 16:11:08.350562 24294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.214.194
--webserver_port=0
--tserver_master_addrs=127.3.214.254:37233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.194
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:08.352046 24294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:08.354051 24294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:08.370834 24299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:08.371954 24300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:08.372030 24302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:08.373212 24294 server_base.cc:1034] running on GCE node
I20260924 16:11:08.374197 24294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:08.375700 24294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:08.376997 24294 hybrid_clock.cc:648] HybridClock initialized: now 1790266268376915 us; error 71 us; skew 500 ppm
I20260924 16:11:08.377523 24294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:11:08.381311 24294 webserver.cc:460] Webserver started at http://127.3.214.194:38227/ using document root <none> and password file <none>
I20260924 16:11:08.382531 24294 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:08.382773 24294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:08.383339 24294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:08.387719 24294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-1/data/instance:
uuid: "d946dcc9e8b24894922eac242587500e"
format_stamp: "Formatted at 2026-09-24 16:11:08 on dist-test-slave-1gc3"
server_key: "36fa20f14057a84ac2ff1ece651afccf"
server_key_iv: "ef03f053d040918fe1e19292b1f47400"
server_key_version: "encryptionkey@0"
I20260924 16:11:08.388978 24294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance:
uuid: "d946dcc9e8b24894922eac242587500e"
format_stamp: "Formatted at 2026-09-24 16:11:08 on dist-test-slave-1gc3"
server_key: "36fa20f14057a84ac2ff1ece651afccf"
server_key_iv: "ef03f053d040918fe1e19292b1f47400"
server_key_version: "encryptionkey@0"
I20260924 16:11:08.396917 24294 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260924 16:11:08.402343 24308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:08.404418 24294 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:11:08.404707 24294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-1/wal
uuid: "d946dcc9e8b24894922eac242587500e"
format_stamp: "Formatted at 2026-09-24 16:11:08 on dist-test-slave-1gc3"
server_key: "36fa20f14057a84ac2ff1ece651afccf"
server_key_iv: "ef03f053d040918fe1e19292b1f47400"
server_key_version: "encryptionkey@0"
I20260924 16:11:08.405023 24294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:08.431073 24294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:08.432581 24294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:08.433014 24294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:08.434432 24294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:11:08.436718 24294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:08.436837 24294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:08.436959 24294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:08.437026 24294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:08.480790 24294 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.194:33509
I20260924 16:11:08.480867 24420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.194:33509 every 8 connection(s)
I20260924 16:11:08.482666 24294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-1/data/info.pb
I20260924 16:11:08.491369 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 24294
I20260924 16:11:08.491585 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-1/wal/instance
I20260924 16:11:08.492246 3931 external_mini_cluster.cc:1428] Setting key 1cd00adb6a7d8260e8d534e44f30d6e5
I20260924 16:11:08.496960 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.214.195:0
--local_ip_for_outbound_sockets=127.3.214.195
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:37233
--builtin_ntp_servers=127.3.214.212:36369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:11:08.502563 24423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:08.486347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.194:47501 (local address 127.3.214.254:37233)
0924 16:11:08.486960 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:08.486966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:08.488472 (+ 1506us) server_negotiation.cc:407] Connection header received
0924 16:11:08.491126 (+ 2654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:08.491266 (+ 140us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:08.491417 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:08.491604 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:08.493783 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:08.494752 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:08.498245 (+ 3493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:08.498785 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:08.498958 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:08.501392 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:08.501399 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:08.501403 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:08.501428 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:08.501493 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:08.501510 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:08.501619 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:08.502036 (+ 417us) server_negotiation.cc:299] Negotiation successful
0924 16:11:08.502215 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":324,"threads_started":1}
I20260924 16:11:08.504395 24424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:08.486818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:37233 (local address 127.3.214.194:47501)
0924 16:11:08.488224 (+ 1406us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:08.488281 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:08.490583 (+ 2302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:08.492044 (+ 1461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:08.492063 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:08.492254 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:08.493510 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:08.493541 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:08.494968 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:08.494972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:08.497838 (+ 2866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:08.497873 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:08.499200 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:08.499209 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:08.499649 (+ 440us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:08.500758 (+ 1109us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:11:08.500762 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:08.500845 (+ 83us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:11:08.501008 (+ 163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:08.501651 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:08.501662 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:08.501672 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 16:11:08.501951 (+ 279us) client_negotiation.cc:240] Negotiation successful
0924 16:11:08.502629 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":420,"threads_started":1}
I20260924 16:11:08.506394 24421 heartbeater.cc:347] Connected to a master server at 127.3.214.254:37233
I20260924 16:11:08.506788 24421 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:08.507699 24421 heartbeater.cc:511] Master 127.3.214.254:37233 requested a full tablet report, sending...
I20260924 16:11:08.509419 24104 ts_manager.cc:194] Registered new tserver with Master: d946dcc9e8b24894922eac242587500e (127.3.214.194:33509)
I20260924 16:11:08.510777 24104 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.214.194:47501
W20260924 16:11:08.868836 24425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:08.869318 24425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:08.869454 24425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:08.877245 24425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:08.877383 24425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:08.877532 24425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.214.195
I20260924 16:11:08.887588 24425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:36369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.214.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.214.195
--webserver_port=0
--tserver_master_addrs=127.3.214.254:37233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.214.195
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:08.888954 24425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:08.890926 24425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:08.906497 24430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:08.906433 24433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:08.906395 24431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:08.907510 24425 server_base.cc:1034] running on GCE node
I20260924 16:11:08.908237 24425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:08.909373 24425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:08.910615 24425 hybrid_clock.cc:648] HybridClock initialized: now 1790266268910517 us; error 84 us; skew 500 ppm
I20260924 16:11:08.911044 24425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:11:08.913718 24425 webserver.cc:460] Webserver started at http://127.3.214.195:42703/ using document root <none> and password file <none>
I20260924 16:11:08.914537 24425 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:08.914644 24425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:08.914973 24425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:08.918092 24425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-2/data/instance:
uuid: "b5c487c84807434db8e55651ff680ff5"
format_stamp: "Formatted at 2026-09-24 16:11:08 on dist-test-slave-1gc3"
server_key: "b417867d9c97824e813083492035c967"
server_key_iv: "75853be80a726642d87208a5e7f4ee8c"
server_key_version: "encryptionkey@0"
I20260924 16:11:08.918962 24425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance:
uuid: "b5c487c84807434db8e55651ff680ff5"
format_stamp: "Formatted at 2026-09-24 16:11:08 on dist-test-slave-1gc3"
server_key: "b417867d9c97824e813083492035c967"
server_key_iv: "75853be80a726642d87208a5e7f4ee8c"
server_key_version: "encryptionkey@0"
I20260924 16:11:08.924954 24425 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 16:11:08.929492 24439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:08.931722 24425 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:11:08.931977 24425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-2/wal
uuid: "b5c487c84807434db8e55651ff680ff5"
format_stamp: "Formatted at 2026-09-24 16:11:08 on dist-test-slave-1gc3"
server_key: "b417867d9c97824e813083492035c967"
server_key_iv: "75853be80a726642d87208a5e7f4ee8c"
server_key_version: "encryptionkey@0"
I20260924 16:11:08.932253 24425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:08.956946 24425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:08.958178 24425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:08.958559 24425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:08.959828 24425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:11:08.961853 24425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:08.961968 24425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:08.962085 24425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:08.962193 24425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 16:11:08.981097 24290 heartbeater.cc:503] Master 127.3.214.254:37233 was elected leader, sending a full tablet report...
I20260924 16:11:09.000190 24425 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.195:43035
I20260924 16:11:09.000272 24551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.195:43035 every 8 connection(s)
I20260924 16:11:09.001895 24425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-2/data/info.pb
I20260924 16:11:09.008152 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 24425
I20260924 16:11:09.008352 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-2/wal/instance
I20260924 16:11:09.008947 3931 external_mini_cluster.cc:1428] Setting key 9e3dac57b6bda864ab1aa9630a1fe34d
I20260924 16:11:09.017334 24554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.005024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:36883 (local address 127.3.214.254:37233)
0924 16:11:09.005690 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:09.005699 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:09.007275 (+ 1576us) server_negotiation.cc:407] Connection header received
0924 16:11:09.009837 (+ 2562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:09.009844 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:09.009965 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:09.010139 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:09.011688 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.012697 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.014404 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.014893 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.015015 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:09.016475 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:09.016480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:09.016484 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:09.016510 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:09.016561 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:09.016574 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:09.016648 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:09.016843 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 16:11:09.016951 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":305,"threads_started":1}
I20260924 16:11:09.018381 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.005835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:37233 (local address 127.3.214.195:36883)
0924 16:11:09.007080 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:09.007128 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:09.009453 (+ 2325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:09.010492 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:09.010504 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:09.010621 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:09.011459 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:09.011480 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.012852 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.012856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:09.014249 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:09.014264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.015130 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.015136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:09.015367 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:09.016080 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:11:09.016083 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:09.016128 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:11:09.016228 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:09.016657 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:09.016663 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:09.016669 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 16:11:09.016809 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 16:11:09.017183 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":359,"threads_started":1}
I20260924 16:11:09.019946 24552 heartbeater.cc:347] Connected to a master server at 127.3.214.254:37233
I20260924 16:11:09.020277 24552 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:09.021080 24552 heartbeater.cc:511] Master 127.3.214.254:37233 requested a full tablet report, sending...
I20260924 16:11:09.022552 24104 ts_manager.cc:194] Registered new tserver with Master: b5c487c84807434db8e55651ff680ff5 (127.3.214.195:43035)
I20260924 16:11:09.023448 24104 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.214.195:36883
I20260924 16:11:09.025707 3931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:11:09.038035 24554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.030783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49484 (local address 127.3.214.254:37233)
0924 16:11:09.030972 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:09.030977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:09.031533 (+ 556us) server_negotiation.cc:407] Connection header received
0924 16:11:09.031634 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:09.031639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:09.031731 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:09.031855 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:09.032444 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.033327 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.035008 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.035519 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.035708 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:09.037218 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:09.037225 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:09.037229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:09.037255 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:09.037299 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:09.037315 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:09.037399 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:09.037544 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 16:11:09.037682 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:11:09.043625 24104 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:11:09.051891 24104 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:11:09.096205 24566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.068240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:33509 (local address 127.0.0.1:50814)
0924 16:11:09.086291 (+ 18051us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:09.086309 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:09.086440 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:09.087806 (+ 1366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:09.087814 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:09.087848 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:09.088296 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:09.088310 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.090412 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.090417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:09.092931 (+ 2514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:09.092950 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.095156 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.095163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:09.095390 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:09.095458 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 16:11:09.095600 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 16:11:09.095824 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17896,"thread_start_us":17142,"threads_started":1}
I20260924 16:11:09.097345 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.067969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:40899 (local address 127.0.0.1:46588)
0924 16:11:09.085838 (+ 17869us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:09.085884 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:09.086092 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:09.089789 (+ 3697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:09.089795 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:09.089828 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:09.090405 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:09.090417 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.092401 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.092407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:09.094377 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:09.094391 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.096581 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.096588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:09.096767 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:09.096802 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 16:11:09.096878 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 16:11:09.097024 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17603,"mutex_wait_us":3,"thread_start_us":18023,"threads_started":1}
I20260924 16:11:09.097694 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.068563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46588 (local address 127.3.214.193:40899)
0924 16:11:09.089062 (+ 20499us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:09.089073 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:09.089111 (+ 38us) server_negotiation.cc:407] Connection header received
0924 16:11:09.089221 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:09.089228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:09.089474 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:09.089817 (+ 343us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:09.090645 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.092251 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.094615 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.096082 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.096232 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:09.096841 (+ 609us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:09.096958 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 16:11:09.097146 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20258,"spinlock_wait_cycles":9472,"thread_start_us":20243,"threads_started":1}
I20260924 16:11:09.097733 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.068963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50814 (local address 127.3.214.194:33509)
0924 16:11:09.085122 (+ 16159us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:09.085148 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:09.086581 (+ 1433us) server_negotiation.cc:407] Connection header received
0924 16:11:09.086762 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:09.086772 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:09.087177 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:09.087445 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:09.088546 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.090214 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.093193 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.094918 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.095093 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:09.096956 (+ 1863us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:09.097051 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 16:11:09.097310 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15942,"thread_start_us":15764,"threads_started":1}
I20260924 16:11:09.100656 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.067523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.195:43035 (local address 127.0.0.1:35262)
0924 16:11:09.080943 (+ 13420us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:09.081030 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:09.081387 (+ 357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:09.089498 (+ 8111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:09.089504 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:09.089546 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:09.090150 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:09.090173 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.092266 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.092273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:09.094952 (+ 2679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:09.094968 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.097836 (+ 2868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.097842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:09.099999 (+ 2157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:09.100037 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 16:11:09.100150 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 16:11:09.100305 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12859,"mutex_wait_us":118,"thread_start_us":14199,"threads_started":1}
I20260924 16:11:09.101027 24567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.067785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35262 (local address 127.3.214.195:43035)
0924 16:11:09.088662 (+ 20877us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:09.088675 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:09.088747 (+ 72us) server_negotiation.cc:407] Connection header received
0924 16:11:09.088925 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:09.088934 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:09.089215 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:09.089454 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:09.090424 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.092091 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.095211 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.097208 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.097370 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:09.097674 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:09.100317 (+ 2643us) server_negotiation.cc:299] Negotiation successful
0924 16:11:09.100573 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20604,"thread_start_us":20398,"threads_started":1}
I20260924 16:11:09.102417 24356 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9066c183a648f2b732793848f2333e (DEFAULT_TABLE table=test-table [id=3604915d02074f7c803b5bb5c58bba48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:09.103421 24225 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9066c183a648f2b732793848f2333e (DEFAULT_TABLE table=test-table [id=3604915d02074f7c803b5bb5c58bba48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:09.104174 24356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9066c183a648f2b732793848f2333e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:09.104915 24225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9066c183a648f2b732793848f2333e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:09.105429 24487 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9066c183a648f2b732793848f2333e (DEFAULT_TABLE table=test-table [id=3604915d02074f7c803b5bb5c58bba48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:09.107197 24487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9066c183a648f2b732793848f2333e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:09.120294 24571 tablet_bootstrap.cc:492] T 6a9066c183a648f2b732793848f2333e P e3e155fda5e4469eb2a7cf4626c4e14b: Bootstrap starting.
I20260924 16:11:09.122995 24572 tablet_bootstrap.cc:492] T 6a9066c183a648f2b732793848f2333e P d946dcc9e8b24894922eac242587500e: Bootstrap starting.
I20260924 16:11:09.125619 24571 tablet_bootstrap.cc:654] T 6a9066c183a648f2b732793848f2333e P e3e155fda5e4469eb2a7cf4626c4e14b: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:09.126945 24573 tablet_bootstrap.cc:492] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5: Bootstrap starting.
I20260924 16:11:09.127260 24572 tablet_bootstrap.cc:654] T 6a9066c183a648f2b732793848f2333e P d946dcc9e8b24894922eac242587500e: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:09.128739 24572 log.cc:824] T 6a9066c183a648f2b732793848f2333e P d946dcc9e8b24894922eac242587500e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:09.128844 24571 log.cc:824] T 6a9066c183a648f2b732793848f2333e P e3e155fda5e4469eb2a7cf4626c4e14b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:09.131031 24573 tablet_bootstrap.cc:654] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:09.132314 24573 log.cc:824] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:09.132678 24571 tablet_bootstrap.cc:492] T 6a9066c183a648f2b732793848f2333e P e3e155fda5e4469eb2a7cf4626c4e14b: No bootstrap required, opened a new log
I20260924 16:11:09.132786 24572 tablet_bootstrap.cc:492] T 6a9066c183a648f2b732793848f2333e P d946dcc9e8b24894922eac242587500e: No bootstrap required, opened a new log
I20260924 16:11:09.133068 24571 ts_tablet_manager.cc:1397] T 6a9066c183a648f2b732793848f2333e P e3e155fda5e4469eb2a7cf4626c4e14b: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260924 16:11:09.133168 24572 ts_tablet_manager.cc:1397] T 6a9066c183a648f2b732793848f2333e P d946dcc9e8b24894922eac242587500e: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260924 16:11:09.136080 24573 tablet_bootstrap.cc:492] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5: No bootstrap required, opened a new log
I20260924 16:11:09.136458 24573 ts_tablet_manager.cc:1397] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260924 16:11:09.142411 24571 raft_consensus.cc:348] T 6a9066c183a648f2b732793848f2333e P e3e155fda5e4469eb2a7cf4626c4e14b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c487c84807434db8e55651ff680ff5" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 43035 } } peers { permanent_uuid: "e3e155fda5e4469eb2a7cf4626c4e14b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 40899 } } peers { permanent_uuid: "d946dcc9e8b24894922eac242587500e" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 33509 } }
I20260924 16:11:09.142930 24571 raft_consensus.cc:374] T 6a9066c183a648f2b732793848f2333e P e3e155fda5e4469eb2a7cf4626c4e14b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:09.143042 24571 raft_consensus.cc:729] T 6a9066c183a648f2b732793848f2333e P e3e155fda5e4469eb2a7cf4626c4e14b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3e155fda5e4469eb2a7cf4626c4e14b, State: Initialized, Role: FOLLOWER
I20260924 16:11:09.143781 24571 consensus_queue.cc:260] T 6a9066c183a648f2b732793848f2333e P e3e155fda5e4469eb2a7cf4626c4e14b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c487c84807434db8e55651ff680ff5" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 43035 } } peers { permanent_uuid: "e3e155fda5e4469eb2a7cf4626c4e14b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 40899 } } peers { permanent_uuid: "d946dcc9e8b24894922eac242587500e" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 33509 } }
I20260924 16:11:09.143909 24572 raft_consensus.cc:348] T 6a9066c183a648f2b732793848f2333e P d946dcc9e8b24894922eac242587500e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c487c84807434db8e55651ff680ff5" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 43035 } } peers { permanent_uuid: "e3e155fda5e4469eb2a7cf4626c4e14b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 40899 } } peers { permanent_uuid: "d946dcc9e8b24894922eac242587500e" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 33509 } }
I20260924 16:11:09.144517 24572 raft_consensus.cc:374] T 6a9066c183a648f2b732793848f2333e P d946dcc9e8b24894922eac242587500e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:09.144621 24572 raft_consensus.cc:729] T 6a9066c183a648f2b732793848f2333e P d946dcc9e8b24894922eac242587500e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d946dcc9e8b24894922eac242587500e, State: Initialized, Role: FOLLOWER
I20260924 16:11:09.145363 24572 consensus_queue.cc:260] T 6a9066c183a648f2b732793848f2333e P d946dcc9e8b24894922eac242587500e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c487c84807434db8e55651ff680ff5" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 43035 } } peers { permanent_uuid: "e3e155fda5e4469eb2a7cf4626c4e14b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 40899 } } peers { permanent_uuid: "d946dcc9e8b24894922eac242587500e" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 33509 } }
I20260924 16:11:09.146422 24571 ts_tablet_manager.cc:1428] T 6a9066c183a648f2b732793848f2333e P e3e155fda5e4469eb2a7cf4626c4e14b: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260924 16:11:09.146917 24421 heartbeater.cc:503] Master 127.3.214.254:37233 was elected leader, sending a full tablet report...
I20260924 16:11:09.147564 24572 ts_tablet_manager.cc:1428] T 6a9066c183a648f2b732793848f2333e P d946dcc9e8b24894922eac242587500e: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260924 16:11:09.148689 24573 raft_consensus.cc:348] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c487c84807434db8e55651ff680ff5" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 43035 } } peers { permanent_uuid: "e3e155fda5e4469eb2a7cf4626c4e14b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 40899 } } peers { permanent_uuid: "d946dcc9e8b24894922eac242587500e" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 33509 } }
I20260924 16:11:09.149115 24573 raft_consensus.cc:374] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:09.149209 24573 raft_consensus.cc:729] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c487c84807434db8e55651ff680ff5, State: Initialized, Role: FOLLOWER
I20260924 16:11:09.149899 24573 consensus_queue.cc:260] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c487c84807434db8e55651ff680ff5" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 43035 } } peers { permanent_uuid: "e3e155fda5e4469eb2a7cf4626c4e14b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 40899 } } peers { permanent_uuid: "d946dcc9e8b24894922eac242587500e" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 33509 } }
I20260924 16:11:09.152638 24552 heartbeater.cc:503] Master 127.3.214.254:37233 was elected leader, sending a full tablet report...
I20260924 16:11:09.154628 24573 ts_tablet_manager.cc:1428] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5: Time spent starting tablet: real 0.018s user 0.013s sys 0.000s
W20260924 16:11:09.203029 24291 tablet.cc:2369] T 6a9066c183a648f2b732793848f2333e P e3e155fda5e4469eb2a7cf4626c4e14b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:11:09.235509 24422 tablet.cc:2369] T 6a9066c183a648f2b732793848f2333e P d946dcc9e8b24894922eac242587500e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:11:09.254136 24553 tablet.cc:2369] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:11:09.262723 24579 raft_consensus.cc:482] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:11:09.263175 24579 raft_consensus.cc:504] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c487c84807434db8e55651ff680ff5" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 43035 } } peers { permanent_uuid: "e3e155fda5e4469eb2a7cf4626c4e14b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 40899 } } peers { permanent_uuid: "d946dcc9e8b24894922eac242587500e" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 33509 } }
I20260924 16:11:09.265457 24579 leader_election.cc:288] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3e155fda5e4469eb2a7cf4626c4e14b (127.3.214.193:40899), d946dcc9e8b24894922eac242587500e (127.3.214.194:33509)
I20260924 16:11:09.271953 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.266413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:57701 (local address 127.3.214.194:33509)
0924 16:11:09.266665 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:09.266670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:09.266690 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:11:09.266770 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:09.266774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:09.266851 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:09.266955 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:09.267543 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.268445 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.270221 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.271007 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.271182 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:09.271259 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:09.271533 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 16:11:09.271668 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260924 16:11:09.271971 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.266035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.194:33509 (local address 127.3.214.195:57701)
0924 16:11:09.266353 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:09.266381 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:09.266555 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:09.267084 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:09.267090 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:09.267131 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:09.267372 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:09.267384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.268591 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.268597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:09.270012 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:09.270030 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.271182 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.271188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:09.271373 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:09.271405 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 16:11:09.271490 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 16:11:09.271665 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260924 16:11:09.273598 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.266271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.214.195:41567 (local address 127.3.214.193:40899)
0924 16:11:09.266480 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:09.266486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:09.267679 (+ 1193us) server_negotiation.cc:407] Connection header received
0924 16:11:09.267843 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:09.267849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:09.267938 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:09.268044 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:09.268790 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.269591 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.271557 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.272610 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.272820 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:09.272897 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:09.273228 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 16:11:09.273371 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 16:11:09.273670 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.266066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:40899 (local address 127.3.214.195:41567)
0924 16:11:09.267555 (+ 1489us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:09.267574 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:09.267711 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:09.268124 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:09.268134 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:09.268171 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:09.268525 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:09.268534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.269773 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.269777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:09.271335 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:09.271351 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.272810 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.272816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:09.273003 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:09.273041 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 16:11:09.273174 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 16:11:09.273326 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1355,"mutex_wait_us":449,"thread_start_us":574,"threads_started":1}
I20260924 16:11:09.273479 24376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9066c183a648f2b732793848f2333e" candidate_uuid: "b5c487c84807434db8e55651ff680ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d946dcc9e8b24894922eac242587500e" is_pre_election: true
I20260924 16:11:09.274165 24376 raft_consensus.cc:2454] T 6a9066c183a648f2b732793848f2333e P d946dcc9e8b24894922eac242587500e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5c487c84807434db8e55651ff680ff5 in term 0.
I20260924 16:11:09.274991 24443 leader_election.cc:302] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5c487c84807434db8e55651ff680ff5, d946dcc9e8b24894922eac242587500e; no voters:
I20260924 16:11:09.275187 24245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9066c183a648f2b732793848f2333e" candidate_uuid: "b5c487c84807434db8e55651ff680ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3e155fda5e4469eb2a7cf4626c4e14b" is_pre_election: true
I20260924 16:11:09.275478 24579 raft_consensus.cc:2781] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:11:09.275702 24579 raft_consensus.cc:482] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:11:09.275820 24579 raft_consensus.cc:3037] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:09.275929 24245 raft_consensus.cc:2454] T 6a9066c183a648f2b732793848f2333e P e3e155fda5e4469eb2a7cf4626c4e14b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5c487c84807434db8e55651ff680ff5 in term 0.
I20260924 16:11:09.278785 24579 raft_consensus.cc:504] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c487c84807434db8e55651ff680ff5" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 43035 } } peers { permanent_uuid: "e3e155fda5e4469eb2a7cf4626c4e14b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 40899 } } peers { permanent_uuid: "d946dcc9e8b24894922eac242587500e" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 33509 } }
I20260924 16:11:09.279541 24579 leader_election.cc:288] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5 [CANDIDATE]: Term 1 election: Requested vote from peers e3e155fda5e4469eb2a7cf4626c4e14b (127.3.214.193:40899), d946dcc9e8b24894922eac242587500e (127.3.214.194:33509)
I20260924 16:11:09.280211 24245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9066c183a648f2b732793848f2333e" candidate_uuid: "b5c487c84807434db8e55651ff680ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3e155fda5e4469eb2a7cf4626c4e14b"
I20260924 16:11:09.280366 24376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9066c183a648f2b732793848f2333e" candidate_uuid: "b5c487c84807434db8e55651ff680ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d946dcc9e8b24894922eac242587500e"
I20260924 16:11:09.280454 24245 raft_consensus.cc:3037] T 6a9066c183a648f2b732793848f2333e P e3e155fda5e4469eb2a7cf4626c4e14b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:09.280646 24376 raft_consensus.cc:3037] T 6a9066c183a648f2b732793848f2333e P d946dcc9e8b24894922eac242587500e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:09.283537 24376 raft_consensus.cc:2454] T 6a9066c183a648f2b732793848f2333e P d946dcc9e8b24894922eac242587500e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5c487c84807434db8e55651ff680ff5 in term 1.
I20260924 16:11:09.283677 24245 raft_consensus.cc:2454] T 6a9066c183a648f2b732793848f2333e P e3e155fda5e4469eb2a7cf4626c4e14b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5c487c84807434db8e55651ff680ff5 in term 1.
I20260924 16:11:09.284265 24443 leader_election.cc:302] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5c487c84807434db8e55651ff680ff5, d946dcc9e8b24894922eac242587500e; no voters:
I20260924 16:11:09.284694 24579 raft_consensus.cc:2781] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:09.285255 24579 raft_consensus.cc:686] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5 [term 1 LEADER]: Becoming Leader. State: Replica: b5c487c84807434db8e55651ff680ff5, State: Running, Role: LEADER
I20260924 16:11:09.285934 24579 consensus_queue.cc:237] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c487c84807434db8e55651ff680ff5" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 43035 } } peers { permanent_uuid: "e3e155fda5e4469eb2a7cf4626c4e14b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 40899 } } peers { permanent_uuid: "d946dcc9e8b24894922eac242587500e" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 33509 } }
I20260924 16:11:09.291816 24102 catalog_manager.cc:5495] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5c487c84807434db8e55651ff680ff5 (127.3.214.195). New cstate: current_term: 1 leader_uuid: "b5c487c84807434db8e55651ff680ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5c487c84807434db8e55651ff680ff5" member_type: VOTER last_known_addr { host: "127.3.214.195" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3e155fda5e4469eb2a7cf4626c4e14b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 40899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d946dcc9e8b24894922eac242587500e" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 33509 } health_report { overall_health: UNKNOWN } } }
I20260924 16:11:09.345570 24567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.339030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35270 (local address 127.3.214.195:43035)
0924 16:11:09.339207 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:09.339212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:09.339276 (+ 64us) server_negotiation.cc:407] Connection header received
0924 16:11:09.339522 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:09.339526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:09.339628 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:09.339772 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:09.340268 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.341106 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.343111 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.343564 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.343684 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:09.344259 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:09.344915 (+ 656us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:09.345044 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:09.345244 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 16:11:09.345326 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:11:09.356261 24376 raft_consensus.cc:1261] T 6a9066c183a648f2b732793848f2333e P d946dcc9e8b24894922eac242587500e [term 1 FOLLOWER]: Refusing update from remote peer b5c487c84807434db8e55651ff680ff5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:09.356390 24245 raft_consensus.cc:1261] T 6a9066c183a648f2b732793848f2333e P e3e155fda5e4469eb2a7cf4626c4e14b [term 1 FOLLOWER]: Refusing update from remote peer b5c487c84807434db8e55651ff680ff5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:09.357656 24579 consensus_queue.cc:1035] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d946dcc9e8b24894922eac242587500e" member_type: VOTER last_known_addr { host: "127.3.214.194" port: 33509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:11:09.358204 24581 consensus_queue.cc:1035] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3e155fda5e4469eb2a7cf4626c4e14b" member_type: VOTER last_known_addr { host: "127.3.214.193" port: 40899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:11:09.375555 24584 mvcc.cc:205] Tried to move back new op lower bound from 7332930639269085184 to 7332930639000162304. Current Snapshot: MvccSnapshot[applied={T|T < 7332930639269085184}]
I20260924 16:11:09.393371 24102 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:11:09.393687 24102 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:11:09.397710 24102 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d2195aa518d4de29bcdb7408a0e760b: Sending DeleteTablet for 3 replicas of tablet 6a9066c183a648f2b732793848f2333e
I20260924 16:11:09.399317 24225 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a9066c183a648f2b732793848f2333e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:09 UTC) from {username='slave'} at 127.0.0.1:46588
I20260924 16:11:09.399322 24487 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a9066c183a648f2b732793848f2333e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:09 UTC) from {username='slave'} at 127.0.0.1:35262
I20260924 16:11:09.399565 24356 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a9066c183a648f2b732793848f2333e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:09 UTC) from {username='slave'} at 127.0.0.1:50814
I20260924 16:11:09.404443 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 24163
I20260924 16:11:09.411713 24596 tablet_replica.cc:330] T 6a9066c183a648f2b732793848f2333e P d946dcc9e8b24894922eac242587500e: stopping tablet replica
I20260924 16:11:09.412395 24596 raft_consensus.cc:2229] T 6a9066c183a648f2b732793848f2333e P d946dcc9e8b24894922eac242587500e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:11:09.413101 24596 raft_consensus.cc:2258] T 6a9066c183a648f2b732793848f2333e P d946dcc9e8b24894922eac242587500e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:09.414157 24594 tablet_replica.cc:330] T 6a9066c183a648f2b732793848f2333e P e3e155fda5e4469eb2a7cf4626c4e14b: stopping tablet replica
I20260924 16:11:09.414716 24594 raft_consensus.cc:2229] T 6a9066c183a648f2b732793848f2333e P e3e155fda5e4469eb2a7cf4626c4e14b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:11:09.415361 24594 raft_consensus.cc:2258] T 6a9066c183a648f2b732793848f2333e P e3e155fda5e4469eb2a7cf4626c4e14b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:09.419116 24595 tablet_replica.cc:330] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5: stopping tablet replica
I20260924 16:11:09.419682 24595 raft_consensus.cc:2229] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:11:09.420617 24595 raft_consensus.cc:2258] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:09.418433 24596 ts_tablet_manager.cc:1905] T 6a9066c183a648f2b732793848f2333e P d946dcc9e8b24894922eac242587500e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:09.421831 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.406645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46604 (local address 127.3.214.193:40899)
0924 16:11:09.406904 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:09.406911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:09.409179 (+ 2268us) server_negotiation.cc:407] Connection header received
0924 16:11:09.409433 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:09.409439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:09.409594 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:09.409760 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:09.412517 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.413833 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.416573 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.417311 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.417422 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:09.419805 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:09.419821 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:09.419841 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:09.420817 (+ 976us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:09.420888 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:09.420917 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:09.421024 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:09.421306 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 16:11:09.421451 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260924 16:11:09.422020 24594 ts_tablet_manager.cc:1905] T 6a9066c183a648f2b732793848f2333e P e3e155fda5e4469eb2a7cf4626c4e14b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:09.423671 24285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:09.424340 24595 ts_tablet_manager.cc:1905] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:09.429171 24596 ts_tablet_manager.cc:1918] T 6a9066c183a648f2b732793848f2333e P d946dcc9e8b24894922eac242587500e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:11:09.429396 24596 log.cc:1196] T 6a9066c183a648f2b732793848f2333e P d946dcc9e8b24894922eac242587500e: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-1/wal/wals/6a9066c183a648f2b732793848f2333e
I20260924 16:11:09.429976 24596 ts_tablet_manager.cc:1939] T 6a9066c183a648f2b732793848f2333e P d946dcc9e8b24894922eac242587500e: Deleting consensus metadata
I20260924 16:11:09.432243 24091 catalog_manager.cc:4841] TS d946dcc9e8b24894922eac242587500e (127.3.214.194:33509): tablet 6a9066c183a648f2b732793848f2333e (table test-table [id=3604915d02074f7c803b5bb5c58bba48]) successfully deleted
I20260924 16:11:09.434170 24595 ts_tablet_manager.cc:1918] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:11:09.434439 24595 log.cc:1196] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5: Deleting WAL directory at /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790265920616583-3931-0/minicluster-data/ts-2/wal/wals/6a9066c183a648f2b732793848f2333e
I20260924 16:11:09.435280 24595 ts_tablet_manager.cc:1939] T 6a9066c183a648f2b732793848f2333e P b5c487c84807434db8e55651ff680ff5: Deleting consensus metadata
I20260924 16:11:09.438467 24090 catalog_manager.cc:4841] TS b5c487c84807434db8e55651ff680ff5 (127.3.214.195:43035): tablet 6a9066c183a648f2b732793848f2333e (table test-table [id=3604915d02074f7c803b5bb5c58bba48]) successfully deleted
I20260924 16:11:09.584232 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 24163
W20260924 16:11:09.604705 24089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.214.193:40899 (error 108)
I20260924 16:11:09.605484 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 24294
I20260924 16:11:09.607651 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.606187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:40899 (local address 127.0.0.1:46606)
0924 16:11:09.607219 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:09.607409 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:40899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":897,"thread_start_us":815,"threads_started":1}
W20260924 16:11:09.607895 24089 catalog_manager.cc:4584] TS e3e155fda5e4469eb2a7cf4626c4e14b (127.3.214.193:40899): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a9066c183a648f2b732793848f2333e: Network error: Client connection negotiation failed: client connection to 127.3.214.193:40899: connect: Connection refused (error 111)
I20260924 16:11:09.614691 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.606779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50822 (local address 127.3.214.194:33509)
0924 16:11:09.606975 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:09.606980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:09.607001 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:11:09.607229 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:09.607235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:09.607347 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:09.607438 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:09.607906 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.608746 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.611103 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.611569 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.611686 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:09.613108 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:09.613120 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:09.613140 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:09.613818 (+ 678us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:09.613883 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:09.613918 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:09.614035 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:09.614232 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 16:11:09.614350 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 16:11:09.615641 24416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:09.661617 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.661097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:40899 (local address 127.0.0.1:46608)
0924 16:11:09.661343 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:09.661439 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:40899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260924 16:11:09.662755 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.662343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:40899 (local address 127.0.0.1:46618)
0924 16:11:09.662548 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:09.662622 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:40899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260924 16:11:09.723210 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.722740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:40899 (local address 127.0.0.1:46632)
0924 16:11:09.722947 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:09.723043 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:40899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 16:11:09.724346 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.723958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:40899 (local address 127.0.0.1:46644)
0924 16:11:09.724121 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:09.724199 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:40899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 16:11:09.775242 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 24294
I20260924 16:11:09.803174 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 24425
I20260924 16:11:09.805099 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.804611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:40899 (local address 127.0.0.1:46654)
0924 16:11:09.804813 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:09.804909 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:40899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 16:11:09.806681 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.806003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:40899 (local address 127.0.0.1:46664)
0924 16:11:09.806336 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:09.806464 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:40899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260924 16:11:09.810936 24567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:09.804028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35278 (local address 127.3.214.195:43035)
0924 16:11:09.804340 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:09.804352 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:09.804382 (+ 30us) server_negotiation.cc:407] Connection header received
0924 16:11:09.804485 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:09.804492 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:09.804638 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:09.804788 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:09.805328 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.806381 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.807728 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:09.808196 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:09.808324 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:09.809397 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:09.809406 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:09.809423 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:09.810154 (+ 731us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:09.810206 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:09.810228 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:09.810334 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:09.810465 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 16:11:09.810585 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260924 16:11:09.811841 24547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:09.950130 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 24425
I20260924 16:11:09.970659 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 24073
I20260924 16:11:09.972191 24134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:10.105417 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:10.104644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.193:40899 (local address 127.0.0.1:46672)
0924 16:11:10.105027 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:10.105165 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.193:40899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 16:11:10.105718 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 24073
2026-09-24T16:11:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3441 ms)
[----------] 6 tests from EncryptionPolicyTest (24992 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:10 dist-test-slave-1gc3 krb5kdc[24606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:10 dist-test-slave-1gc3 krb5kdc[24606](info): set up 2 sockets
Sep 24 16:11:10 dist-test-slave-1gc3 krb5kdc[24606](info): commencing operation
krb5kdc: starting...
W20260924 16:11:11.196623 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.003s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:11:11 dist-test-slave-1gc3 krb5kdc[24606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:11:11.523672 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:11.933668 24619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:11.934142 24619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:11.934312 24619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:11.942780 24619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:11.942943 24619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:11.943007 24619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:11.943101 24619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:11.943164 24619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:11.954648 24619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42547
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:11.956544 24619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:11.958292 24619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:11.972630 24625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:11.972817 24627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:11.972638 24624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:11.973409 24619 server_base.cc:1034] running on GCE node
I20260924 16:11:11.974759 24619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:11.976764 24619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:11.978066 24619 hybrid_clock.cc:648] HybridClock initialized: now 1790266271977992 us; error 57 us; skew 500 ppm
I20260924 16:11:11.978686 24619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:11 dist-test-slave-1gc3 krb5kdc[24606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:11.987136 24619 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:11:11.989073 24619 webserver.cc:460] Webserver started at http://127.0.0.1:42085/ using document root <none> and password file <none>
I20260924 16:11:11.989923 24619 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:11.990056 24619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:11.990424 24619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:11.993750 24619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "dc8c398181d94f11be757e3bc2eea3c9"
format_stamp: "Formatted at 2026-09-24 16:11:11 on dist-test-slave-1gc3"
server_key: "328017222933515570997d143046f70c"
server_key_iv: "2aced1f7a7fe3cfb83dc1bb5e67fdf9f"
server_key_version: "encryptionkey@0"
I20260924 16:11:11.994607 24619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "dc8c398181d94f11be757e3bc2eea3c9"
format_stamp: "Formatted at 2026-09-24 16:11:11 on dist-test-slave-1gc3"
server_key: "328017222933515570997d143046f70c"
server_key_iv: "2aced1f7a7fe3cfb83dc1bb5e67fdf9f"
server_key_version: "encryptionkey@0"
I20260924 16:11:12.001045 24619 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260924 16:11:12.005837 24634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:12.007993 24619 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:11:12.008251 24619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "dc8c398181d94f11be757e3bc2eea3c9"
format_stamp: "Formatted at 2026-09-24 16:11:11 on dist-test-slave-1gc3"
server_key: "328017222933515570997d143046f70c"
server_key_iv: "2aced1f7a7fe3cfb83dc1bb5e67fdf9f"
server_key_version: "encryptionkey@0"
I20260924 16:11:12.008523 24619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:12.031632 24619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:12.036360 24619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:12.036706 24619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:12.058485 24619 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42547
I20260924 16:11:12.058548 24685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42547 every 8 connection(s)
I20260924 16:11:12.060171 24619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:12.065557 24686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:12.068428 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 24619
I20260924 16:11:12.068724 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:11:12.069679 3931 external_mini_cluster.cc:1428] Setting key 18aa3d0803197b7f5ab3573e1a6cdd26
I20260924 16:11:12.081041 24686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc8c398181d94f11be757e3bc2eea3c9: Bootstrap starting.
I20260924 16:11:12.085572 24686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc8c398181d94f11be757e3bc2eea3c9: Neither blocks nor log segments found. Creating new log.
Sep 24 16:11:12 dist-test-slave-1gc3 krb5kdc[24606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 16:11:12.087078 24686 log.cc:824] T 00000000000000000000000000000000 P dc8c398181d94f11be757e3bc2eea3c9: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:12.091269 24686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc8c398181d94f11be757e3bc2eea3c9: No bootstrap required, opened a new log
I20260924 16:11:12.096755 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:12.072706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59769 (local address 127.0.0.1:42547)
0924 16:11:12.073846 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:12.073863 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:12.073914 (+ 51us) server_negotiation.cc:407] Connection header received
0924 16:11:12.076059 (+ 2145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:12.076093 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:12.076621 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:12.077072 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:12.078653 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:12.080058 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:12.081347 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:12.082144 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:12.082325 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:12.087267 (+ 4942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:12.087282 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:12.087312 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:12.087369 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:12.091065 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:12.092047 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:12.092056 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:12.092064 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:12.092316 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:12.093366 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:12.093372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:12.093374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:12.094056 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:12.094404 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:12.095089 (+ 685us) server_negotiation.cc:299] Negotiation successful
0924 16:11:12.095478 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":501,"threads_started":1}
I20260924 16:11:12.101783 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:11:12.101677 24686 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc8c398181d94f11be757e3bc2eea3c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc8c398181d94f11be757e3bc2eea3c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42547 } }
I20260924 16:11:12.102272 24686 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc8c398181d94f11be757e3bc2eea3c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:12.102375 24686 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc8c398181d94f11be757e3bc2eea3c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc8c398181d94f11be757e3bc2eea3c9, State: Initialized, Role: FOLLOWER
I20260924 16:11:12.103178 24686 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc8c398181d94f11be757e3bc2eea3c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc8c398181d94f11be757e3bc2eea3c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42547 } }
I20260924 16:11:12.103468 24686 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc8c398181d94f11be757e3bc2eea3c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:12.103609 24686 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc8c398181d94f11be757e3bc2eea3c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:12.103790 24686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc8c398181d94f11be757e3bc2eea3c9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:12.106704 24686 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc8c398181d94f11be757e3bc2eea3c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc8c398181d94f11be757e3bc2eea3c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42547 } }
I20260924 16:11:12.107452 24686 leader_election.cc:302] T 00000000000000000000000000000000 P dc8c398181d94f11be757e3bc2eea3c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc8c398181d94f11be757e3bc2eea3c9; no voters:
I20260924 16:11:12.108426 24686 leader_election.cc:288] T 00000000000000000000000000000000 P dc8c398181d94f11be757e3bc2eea3c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:12.108568 24696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc8c398181d94f11be757e3bc2eea3c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:12.110433 24696 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc8c398181d94f11be757e3bc2eea3c9 [term 1 LEADER]: Becoming Leader. State: Replica: dc8c398181d94f11be757e3bc2eea3c9, State: Running, Role: LEADER
I20260924 16:11:12.111368 24686 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc8c398181d94f11be757e3bc2eea3c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:12.111230 24696 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc8c398181d94f11be757e3bc2eea3c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc8c398181d94f11be757e3bc2eea3c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42547 } }
I20260924 16:11:12.115792 24698 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc8c398181d94f11be757e3bc2eea3c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc8c398181d94f11be757e3bc2eea3c9. Latest consensus state: current_term: 1 leader_uuid: "dc8c398181d94f11be757e3bc2eea3c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc8c398181d94f11be757e3bc2eea3c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42547 } } }
I20260924 16:11:12.116362 24698 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc8c398181d94f11be757e3bc2eea3c9 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:12.117053 24697 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc8c398181d94f11be757e3bc2eea3c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc8c398181d94f11be757e3bc2eea3c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc8c398181d94f11be757e3bc2eea3c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42547 } } }
I20260924 16:11:12.117389 24697 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc8c398181d94f11be757e3bc2eea3c9 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:12.124783 24704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:12.128027 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:12.107445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48883 (local address 127.0.0.1:42547)
0924 16:11:12.107688 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:12.107693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:12.108154 (+ 461us) server_negotiation.cc:407] Connection header received
0924 16:11:12.108434 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:12.108439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:12.108545 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:12.108682 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:12.110994 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:12.112177 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:12.114440 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:12.115200 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:12.115317 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:12.119174 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:12.119183 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:12.119188 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:12.119238 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:12.123583 (+ 4345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:12.124797 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:12.124807 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:12.124810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:12.125000 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:12.125697 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:12.125702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:12.125706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:12.126241 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:12.127210 (+ 969us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:12.127418 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 16:11:12.127625 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260924 16:11:12.132450 24704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:12.144218 24704 catalog_manager.cc:1347] Generated new cluster ID: f46d22c6f0634de0a5bf60ed0a49d9d3
I20260924 16:11:12.144371 24704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:12.167431 24704 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:12.169982 24704 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:12.184875 24704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc8c398181d94f11be757e3bc2eea3c9: Generated new TSK 0
I20260924 16:11:12.185956 24704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:11:12.192732 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 24619
I20260924 16:11:12.194811 24681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:12.301568 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 24619
W20260924 16:11:12.321498 24604 connection.cc:535] client connection to 127.0.0.1:42547 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42547): Connection reset by peer (error 104)
2026-09-24T16:11:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2188 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:12 dist-test-slave-1gc3 krb5kdc[24719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:12 dist-test-slave-1gc3 krb5kdc[24719](info): set up 2 sockets
Sep 24 16:11:12 dist-test-slave-1gc3 krb5kdc[24719](info): commencing operation
krb5kdc: starting...
W20260924 16:11:13.356055 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.006s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:11:13 dist-test-slave-1gc3 krb5kdc[24719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:11:13.669672 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38701
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:14.079077 24732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:14.079486 24732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:14.079627 24732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:14.088032 24732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:14.088209 24732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:14.088265 24732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:14.088373 24732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:14.088428 24732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:14.100459 24732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38701
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:14.102404 24732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:14.104197 24732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:14.119058 24737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:14.118965 24740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:14.119033 24738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:14.119772 24732 server_base.cc:1034] running on GCE node
I20260924 16:11:14.120723 24732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:14.122581 24732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:14.123891 24732 hybrid_clock.cc:648] HybridClock initialized: now 1790266274123787 us; error 54 us; skew 500 ppm
I20260924 16:11:14.124362 24732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:14 dist-test-slave-1gc3 krb5kdc[24719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:14.128118 24732 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:11:14.130076 24732 webserver.cc:460] Webserver started at http://127.0.0.1:45909/ using document root <none> and password file <none>
I20260924 16:11:14.130960 24732 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:14.131093 24732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:14.131469 24732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:14.135078 24732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "85ba5bc9b6a047c4bde979490e89885e"
format_stamp: "Formatted at 2026-09-24 16:11:14 on dist-test-slave-1gc3"
server_key: "a8b6e79351427191103198e1029e97a6"
server_key_iv: "0dfa6e5177733e8d90fff1820584c969"
server_key_version: "encryptionkey@0"
I20260924 16:11:14.135967 24732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "85ba5bc9b6a047c4bde979490e89885e"
format_stamp: "Formatted at 2026-09-24 16:11:14 on dist-test-slave-1gc3"
server_key: "a8b6e79351427191103198e1029e97a6"
server_key_iv: "0dfa6e5177733e8d90fff1820584c969"
server_key_version: "encryptionkey@0"
I20260924 16:11:14.142575 24732 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.006s
I20260924 16:11:14.147454 24747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:14.149427 24732 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:11:14.149670 24732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "85ba5bc9b6a047c4bde979490e89885e"
format_stamp: "Formatted at 2026-09-24 16:11:14 on dist-test-slave-1gc3"
server_key: "a8b6e79351427191103198e1029e97a6"
server_key_iv: "0dfa6e5177733e8d90fff1820584c969"
server_key_version: "encryptionkey@0"
I20260924 16:11:14.149926 24732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:14.177294 24732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:14.182934 24732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:14.183248 24732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:14.204492 24732 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38701
I20260924 16:11:14.204582 24798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38701 every 8 connection(s)
I20260924 16:11:14.206141 24732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:14.211345 24799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:14.214427 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 24732
I20260924 16:11:14.214624 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:11:14.215116 3931 external_mini_cluster.cc:1428] Setting key 829ccdb97b685bbb3a1bb2cb28b4bd8c
I20260924 16:11:14.227398 24799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e: Bootstrap starting.
I20260924 16:11:14.232908 24799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:14.233952 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:14.217706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54587 (local address 127.0.0.1:38701)
0924 16:11:14.218869 (+ 1163us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:14.218891 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:14.218958 (+ 67us) server_negotiation.cc:407] Connection header received
0924 16:11:14.221516 (+ 2558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:14.221560 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:14.222323 (+ 763us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:14.222962 (+ 639us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:14.223662 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:14.225931 (+ 2269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:14.227580 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:14.228725 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:14.228951 (+ 226us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:14.230706 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:14.230729 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:14.230759 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:14.231672 (+ 913us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:14.231823 (+ 151us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:14.231873 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:14.232094 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:14.232541 (+ 447us) server_negotiation.cc:299] Negotiation successful
0924 16:11:14.232736 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":812,"thread_start_us":506,"threads_started":1}
I20260924 16:11:14.234385 24799 log.cc:824] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:14.238286 24799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e: No bootstrap required, opened a new log
I20260924 16:11:14.239212 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:11:14.247347 24799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ba5bc9b6a047c4bde979490e89885e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38701 } }
I20260924 16:11:14.247849 24799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:14.247961 24799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85ba5bc9b6a047c4bde979490e89885e, State: Initialized, Role: FOLLOWER
I20260924 16:11:14.248602 24799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ba5bc9b6a047c4bde979490e89885e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38701 } }
I20260924 16:11:14.248843 24799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:14.248992 24799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:14.249178 24799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:14.252127 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:14.244457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44181 (local address 127.0.0.1:38701)
0924 16:11:14.244727 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:14.244735 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:14.245118 (+ 383us) server_negotiation.cc:407] Connection header received
0924 16:11:14.245332 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:14.245339 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:14.245435 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:14.245605 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:14.246272 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:14.247377 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:14.249111 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:14.249761 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:14.249930 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:14.251215 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:14.251222 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:14.251227 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:14.251250 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:14.251321 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:14.251349 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:14.251471 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:14.251712 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 16:11:14.251771 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 16:11:14.252229 24799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ba5bc9b6a047c4bde979490e89885e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38701 } }
I20260924 16:11:14.252974 24799 leader_election.cc:302] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85ba5bc9b6a047c4bde979490e89885e; no voters:
I20260924 16:11:14.253670 24799 leader_election.cc:288] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:14.253921 24809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:14.256068 24799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:14.257913 24809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e [term 1 LEADER]: Becoming Leader. State: Replica: 85ba5bc9b6a047c4bde979490e89885e, State: Running, Role: LEADER
I20260924 16:11:14.258852 24809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ba5bc9b6a047c4bde979490e89885e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38701 } }
I20260924 16:11:14.263033 24810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85ba5bc9b6a047c4bde979490e89885e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ba5bc9b6a047c4bde979490e89885e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38701 } } }
I20260924 16:11:14.263761 24810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:14.263628 24811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85ba5bc9b6a047c4bde979490e89885e. Latest consensus state: current_term: 1 leader_uuid: "85ba5bc9b6a047c4bde979490e89885e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ba5bc9b6a047c4bde979490e89885e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38701 } } }
I20260924 16:11:14.264209 24811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e [sys.catalog]: This master's current role is: LEADER
W20260924 16:11:14.268970 24823 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:11:14.269135 24823 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:11:14.269269 24824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:14.271296 24824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:14.282399 24824 catalog_manager.cc:1347] Generated new cluster ID: 0c4300444642437f85d24a7fed61f2e0
I20260924 16:11:14.282574 24824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:14.323757 24824 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:14.325776 24824 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:14.337673 24824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85ba5bc9b6a047c4bde979490e89885e: Generated new TSK 0
I20260924 16:11:14.338936 24824 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:11:14.351236 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 24732
W20260924 16:11:14.353168 24794 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54587
W20260924 16:11:14.354188 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 24732: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:11:14.354511 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 24732
W20260924 16:11:14.371169 24717 connection.cc:535] client connection to 127.0.0.1:38701 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38701): Connection reset by peer (error 104)
2026-09-24T16:11:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2048 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:14 dist-test-slave-1gc3 krb5kdc[24830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:14 dist-test-slave-1gc3 krb5kdc[24830](info): set up 2 sockets
Sep 24 16:11:14 dist-test-slave-1gc3 krb5kdc[24830](info): commencing operation
krb5kdc: starting...
W20260924 16:11:15.424129 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:11:15 dist-test-slave-1gc3 krb5kdc[24830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:11:15.747363 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44341
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:16.155663 24843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:16.156085 24843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:16.156241 24843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:16.164426 24843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:16.164608 24843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:16.164693 24843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:16.164783 24843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:16.164857 24843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:16.176470 24843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44341
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:16.178380 24843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:16.180177 24843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:16.195604 24849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:16.195667 24848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:16.195683 24851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:16.197645 24843 server_base.cc:1034] running on GCE node
I20260924 16:11:16.198871 24843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:16.200928 24843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:16.202224 24843 hybrid_clock.cc:648] HybridClock initialized: now 1790266276202170 us; error 50 us; skew 500 ppm
I20260924 16:11:16.202666 24843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:16 dist-test-slave-1gc3 krb5kdc[24830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:16.206785 24843 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:11:16.208770 24843 webserver.cc:460] Webserver started at http://127.0.0.1:34129/ using document root <none> and password file <none>
I20260924 16:11:16.209623 24843 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:16.209760 24843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:16.210146 24843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:16.213887 24843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "554afe726ba6450c8e3264c99291b12e"
format_stamp: "Formatted at 2026-09-24 16:11:16 on dist-test-slave-1gc3"
server_key: "1c91305e5b33ab6327f20a452db86427"
server_key_iv: "f80b191f0b6e873d53dafea061e8432f"
server_key_version: "encryptionkey@0"
I20260924 16:11:16.215075 24843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "554afe726ba6450c8e3264c99291b12e"
format_stamp: "Formatted at 2026-09-24 16:11:16 on dist-test-slave-1gc3"
server_key: "1c91305e5b33ab6327f20a452db86427"
server_key_iv: "f80b191f0b6e873d53dafea061e8432f"
server_key_version: "encryptionkey@0"
I20260924 16:11:16.221953 24843 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 16:11:16.226857 24858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:16.228793 24843 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:11:16.229043 24843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "554afe726ba6450c8e3264c99291b12e"
format_stamp: "Formatted at 2026-09-24 16:11:16 on dist-test-slave-1gc3"
server_key: "1c91305e5b33ab6327f20a452db86427"
server_key_iv: "f80b191f0b6e873d53dafea061e8432f"
server_key_version: "encryptionkey@0"
I20260924 16:11:16.229271 24843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:16.236016 24843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:16.241569 24843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:16.241859 24843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:16.263518 24843 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44341
I20260924 16:11:16.263613 24909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44341 every 8 connection(s)
I20260924 16:11:16.265120 24843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:16.270165 24910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:16.271337 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 24843
I20260924 16:11:16.271540 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:11:16.272014 3931 external_mini_cluster.cc:1428] Setting key 36bb1a74711981490dd8206f07924e0d
I20260924 16:11:16.281713 24913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:16.274155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51027 (local address 127.0.0.1:44341)
0924 16:11:16.275258 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:16.275274 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:16.275325 (+ 51us) server_negotiation.cc:407] Connection header received
0924 16:11:16.277712 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:16.277749 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:16.278266 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:16.278708 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:16.279862 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:16.279870 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:16.279887 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:16.279903 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:16.279981 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:16.280009 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:16.280253 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:16.280403 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 16:11:16.280655 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":459,"threads_started":1}
I20260924 16:11:16.286527 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:11:16.286655 24910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554afe726ba6450c8e3264c99291b12e: Bootstrap starting.
I20260924 16:11:16.290825 24910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 554afe726ba6450c8e3264c99291b12e: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:16.292559 24910 log.cc:824] T 00000000000000000000000000000000 P 554afe726ba6450c8e3264c99291b12e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:16.294862 24913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:16.291527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41847 (local address 127.0.0.1:44341)
0924 16:11:16.291758 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:16.291764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:16.292191 (+ 427us) server_negotiation.cc:407] Connection header received
0924 16:11:16.292507 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:16.292512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:16.292585 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:16.292705 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:16.294127 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:16.294134 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:16.294140 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:16.294165 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:16.294222 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:16.294245 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:16.294378 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:16.294517 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 16:11:16.294574 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 16:11:16.296945 24910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554afe726ba6450c8e3264c99291b12e: No bootstrap required, opened a new log
I20260924 16:11:16.305774 24910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 554afe726ba6450c8e3264c99291b12e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554afe726ba6450c8e3264c99291b12e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } }
I20260924 16:11:16.306396 24910 raft_consensus.cc:374] T 00000000000000000000000000000000 P 554afe726ba6450c8e3264c99291b12e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:16.306499 24910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 554afe726ba6450c8e3264c99291b12e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554afe726ba6450c8e3264c99291b12e, State: Initialized, Role: FOLLOWER
I20260924 16:11:16.307268 24910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 554afe726ba6450c8e3264c99291b12e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554afe726ba6450c8e3264c99291b12e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } }
I20260924 16:11:16.307539 24910 raft_consensus.cc:388] T 00000000000000000000000000000000 P 554afe726ba6450c8e3264c99291b12e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:16.307690 24910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 554afe726ba6450c8e3264c99291b12e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:16.307868 24910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 554afe726ba6450c8e3264c99291b12e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:16.310550 24910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 554afe726ba6450c8e3264c99291b12e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554afe726ba6450c8e3264c99291b12e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } }
I20260924 16:11:16.311229 24910 leader_election.cc:302] T 00000000000000000000000000000000 P 554afe726ba6450c8e3264c99291b12e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 554afe726ba6450c8e3264c99291b12e; no voters:
I20260924 16:11:16.312000 24910 leader_election.cc:288] T 00000000000000000000000000000000 P 554afe726ba6450c8e3264c99291b12e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:16.312206 24920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 554afe726ba6450c8e3264c99291b12e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:16.313118 24920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 554afe726ba6450c8e3264c99291b12e [term 1 LEADER]: Becoming Leader. State: Replica: 554afe726ba6450c8e3264c99291b12e, State: Running, Role: LEADER
I20260924 16:11:16.313774 24920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 554afe726ba6450c8e3264c99291b12e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554afe726ba6450c8e3264c99291b12e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } }
I20260924 16:11:16.314718 24910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 554afe726ba6450c8e3264c99291b12e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:16.316686 24921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554afe726ba6450c8e3264c99291b12e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "554afe726ba6450c8e3264c99291b12e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554afe726ba6450c8e3264c99291b12e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } } }
I20260924 16:11:16.317672 24921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554afe726ba6450c8e3264c99291b12e [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:16.317471 24922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554afe726ba6450c8e3264c99291b12e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 554afe726ba6450c8e3264c99291b12e. Latest consensus state: current_term: 1 leader_uuid: "554afe726ba6450c8e3264c99291b12e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554afe726ba6450c8e3264c99291b12e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } } }
I20260924 16:11:16.317966 24922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554afe726ba6450c8e3264c99291b12e [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:16.331527 24928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:16.337221 24928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:16.348968 24928 catalog_manager.cc:1347] Generated new cluster ID: b41183f54950494e834a1086519f2467
I20260924 16:11:16.349148 24928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:16.372949 24928 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:16.373518 24928 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:16.385581 24928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 554afe726ba6450c8e3264c99291b12e: Generated new TSK 0
I20260924 16:11:16.386677 24928 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:11:16.395605 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 24843
W20260924 16:11:16.398461 24905 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51027
W20260924 16:11:16.399374 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 24843: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:11:16.399523 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 24843
2026-09-24T16:11:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2043 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:16 dist-test-slave-1gc3 krb5kdc[24941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:16 dist-test-slave-1gc3 krb5kdc[24941](info): set up 2 sockets
Sep 24 16:11:16 dist-test-slave-1gc3 krb5kdc[24941](info): commencing operation
krb5kdc: starting...
W20260924 16:11:17.456493 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:11:17 dist-test-slave-1gc3 krb5kdc[24941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:11:17.773597 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36287
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:18.170183 24954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:18.170650 24954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:18.170801 24954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:18.179222 24954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:18.179425 24954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:18.179518 24954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:18.179610 24954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:18.179672 24954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:18.191803 24954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:18.193766 24954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:18.195689 24954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:18.211329 24959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:18.211647 24960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:18.212011 24962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:18.212988 24954 server_base.cc:1034] running on GCE node
I20260924 16:11:18.214514 24954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:18.216832 24954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:18.218255 24954 hybrid_clock.cc:648] HybridClock initialized: now 1790266278218136 us; error 54 us; skew 500 ppm
I20260924 16:11:18.218797 24954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:18 dist-test-slave-1gc3 krb5kdc[24941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:18.227164 24954 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:11:18.229717 24954 webserver.cc:460] Webserver started at http://127.0.0.1:37591/ using document root <none> and password file <none>
I20260924 16:11:18.230809 24954 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:18.230969 24954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:18.231391 24954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:18.235491 24954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "c461dfc34a474e3ab7dbb9a204e3d832"
format_stamp: "Formatted at 2026-09-24 16:11:18 on dist-test-slave-1gc3"
server_key: "71dfd5626519dfaa86cf176f1a281fe7"
server_key_iv: "51b0f138694fe03800b43ef3dcbd1bb0"
server_key_version: "encryptionkey@0"
I20260924 16:11:18.236496 24954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "c461dfc34a474e3ab7dbb9a204e3d832"
format_stamp: "Formatted at 2026-09-24 16:11:18 on dist-test-slave-1gc3"
server_key: "71dfd5626519dfaa86cf176f1a281fe7"
server_key_iv: "51b0f138694fe03800b43ef3dcbd1bb0"
server_key_version: "encryptionkey@0"
I20260924 16:11:18.244232 24954 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260924 16:11:18.250588 24969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:18.252920 24954 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260924 16:11:18.253248 24954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "c461dfc34a474e3ab7dbb9a204e3d832"
format_stamp: "Formatted at 2026-09-24 16:11:18 on dist-test-slave-1gc3"
server_key: "71dfd5626519dfaa86cf176f1a281fe7"
server_key_iv: "51b0f138694fe03800b43ef3dcbd1bb0"
server_key_version: "encryptionkey@0"
I20260924 16:11:18.253531 24954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:18.283557 24954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:18.289525 24954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:18.289921 24954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:18.313695 25020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36287 every 8 connection(s)
I20260924 16:11:18.313735 24954 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36287
I20260924 16:11:18.315749 24954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:18.319561 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 24954
I20260924 16:11:18.319790 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:11:18.320317 3931 external_mini_cluster.cc:1428] Setting key 5bf5ff484f33f580ace53d45300235cd
I20260924 16:11:18.320812 25021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:18.337725 25021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c461dfc34a474e3ab7dbb9a204e3d832: Bootstrap starting.
Sep 24 16:11:18 dist-test-slave-1gc3 krb5kdc[24941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 16:11:18.341712 25021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c461dfc34a474e3ab7dbb9a204e3d832: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:18.343186 25021 log.cc:824] T 00000000000000000000000000000000 P c461dfc34a474e3ab7dbb9a204e3d832: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:18.347297 25021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c461dfc34a474e3ab7dbb9a204e3d832: No bootstrap required, opened a new log
I20260924 16:11:18.350903 25024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:18.323208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46109 (local address 127.0.0.1:36287)
0924 16:11:18.324334 (+ 1126us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:18.324354 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:18.324418 (+ 64us) server_negotiation.cc:407] Connection header received
0924 16:11:18.327080 (+ 2662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:18.327128 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:18.327877 (+ 749us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:18.328477 (+ 600us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:18.330320 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:18.332363 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:18.333991 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:18.335257 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:18.335509 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:18.340869 (+ 5360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:18.340884 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:18.340906 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:18.340964 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:18.345065 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:18.346346 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:18.346361 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:18.346371 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:18.346623 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:18.347163 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:18.347169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:18.347172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:18.347944 (+ 772us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:18.348304 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:18.348829 (+ 525us) server_negotiation.cc:299] Negotiation successful
0924 16:11:18.349370 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":395,"threads_started":1}
I20260924 16:11:18.356685 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:11:18.357589 25021 raft_consensus.cc:348] T 00000000000000000000000000000000 P c461dfc34a474e3ab7dbb9a204e3d832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c461dfc34a474e3ab7dbb9a204e3d832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36287 } }
I20260924 16:11:18.358039 25021 raft_consensus.cc:374] T 00000000000000000000000000000000 P c461dfc34a474e3ab7dbb9a204e3d832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:18.358160 25021 raft_consensus.cc:729] T 00000000000000000000000000000000 P c461dfc34a474e3ab7dbb9a204e3d832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c461dfc34a474e3ab7dbb9a204e3d832, State: Initialized, Role: FOLLOWER
I20260924 16:11:18.358848 25021 consensus_queue.cc:260] T 00000000000000000000000000000000 P c461dfc34a474e3ab7dbb9a204e3d832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c461dfc34a474e3ab7dbb9a204e3d832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36287 } }
I20260924 16:11:18.359104 25021 raft_consensus.cc:388] T 00000000000000000000000000000000 P c461dfc34a474e3ab7dbb9a204e3d832 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:18.359246 25021 raft_consensus.cc:482] T 00000000000000000000000000000000 P c461dfc34a474e3ab7dbb9a204e3d832 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:18.359398 25021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c461dfc34a474e3ab7dbb9a204e3d832 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:18.362013 25021 raft_consensus.cc:504] T 00000000000000000000000000000000 P c461dfc34a474e3ab7dbb9a204e3d832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c461dfc34a474e3ab7dbb9a204e3d832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36287 } }
I20260924 16:11:18.362852 25021 leader_election.cc:302] T 00000000000000000000000000000000 P c461dfc34a474e3ab7dbb9a204e3d832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c461dfc34a474e3ab7dbb9a204e3d832; no voters:
I20260924 16:11:18.363620 25021 leader_election.cc:288] T 00000000000000000000000000000000 P c461dfc34a474e3ab7dbb9a204e3d832 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:18.363745 25031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c461dfc34a474e3ab7dbb9a204e3d832 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:18.366990 25021 sys_catalog.cc:564] T 00000000000000000000000000000000 P c461dfc34a474e3ab7dbb9a204e3d832 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:18.367137 25031 raft_consensus.cc:686] T 00000000000000000000000000000000 P c461dfc34a474e3ab7dbb9a204e3d832 [term 1 LEADER]: Becoming Leader. State: Replica: c461dfc34a474e3ab7dbb9a204e3d832, State: Running, Role: LEADER
I20260924 16:11:18.367908 25031 consensus_queue.cc:237] T 00000000000000000000000000000000 P c461dfc34a474e3ab7dbb9a204e3d832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c461dfc34a474e3ab7dbb9a204e3d832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36287 } }
I20260924 16:11:18.371526 25032 sys_catalog.cc:455] T 00000000000000000000000000000000 P c461dfc34a474e3ab7dbb9a204e3d832 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c461dfc34a474e3ab7dbb9a204e3d832. Latest consensus state: current_term: 1 leader_uuid: "c461dfc34a474e3ab7dbb9a204e3d832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c461dfc34a474e3ab7dbb9a204e3d832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36287 } } }
I20260924 16:11:18.372035 25032 sys_catalog.cc:458] T 00000000000000000000000000000000 P c461dfc34a474e3ab7dbb9a204e3d832 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:18.376653 25034 sys_catalog.cc:455] T 00000000000000000000000000000000 P c461dfc34a474e3ab7dbb9a204e3d832 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c461dfc34a474e3ab7dbb9a204e3d832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c461dfc34a474e3ab7dbb9a204e3d832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36287 } } }
I20260924 16:11:18.382692 25034 sys_catalog.cc:458] T 00000000000000000000000000000000 P c461dfc34a474e3ab7dbb9a204e3d832 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:18.384018 25039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:18.384564 25024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:18.362727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36327 (local address 127.0.0.1:36287)
0924 16:11:18.362954 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:18.362962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:18.363500 (+ 538us) server_negotiation.cc:407] Connection header received
0924 16:11:18.363755 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:18.363761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:18.363858 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:18.363964 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:18.366815 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:18.368067 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:18.370159 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:18.370916 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:18.371067 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:18.375374 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:18.375383 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:18.375389 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:18.375453 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:18.379982 (+ 4529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:18.381268 (+ 1286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:18.381275 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:18.381278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:18.381476 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:18.381976 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:18.381982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:18.381988 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:18.382559 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:18.382833 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:18.383506 (+ 673us) server_negotiation.cc:299] Negotiation successful
0924 16:11:18.383760 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 16:11:18.392544 25039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:18.405972 25039 catalog_manager.cc:1347] Generated new cluster ID: b88cca85e37243dc9f0d1dcc567bd3c5
I20260924 16:11:18.406213 25039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:18.424787 25039 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:18.427076 25039 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:18.447481 25039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c461dfc34a474e3ab7dbb9a204e3d832: Generated new TSK 0
I20260924 16:11:18.448869 25039 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:11:18.466903 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 24954
I20260924 16:11:18.468628 25016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:18.570274 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 24954
2026-09-24T16:11:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2175 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:18 dist-test-slave-1gc3 krb5kdc[25055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:18 dist-test-slave-1gc3 krb5kdc[25055](info): set up 2 sockets
Sep 24 16:11:18 dist-test-slave-1gc3 krb5kdc[25055](info): commencing operation
krb5kdc: starting...
W20260924 16:11:19.648470 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.006s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:11:19 dist-test-slave-1gc3 krb5kdc[25055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:11:19.982561 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44913
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:20.400416 25068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:20.400871 25068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:20.401023 25068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:20.409509 25068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:20.409708 25068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:20.409787 25068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:20.409879 25068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:20.409956 25068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:20.421703 25068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:20.423578 25068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:20.425235 25068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:20.439546 25074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:20.439625 25076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:20.439639 25073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:20.440982 25068 server_base.cc:1034] running on GCE node
I20260924 16:11:20.442211 25068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:20.444299 25068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:20.445591 25068 hybrid_clock.cc:648] HybridClock initialized: now 1790266280445529 us; error 56 us; skew 500 ppm
I20260924 16:11:20.446200 25068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:20 dist-test-slave-1gc3 krb5kdc[25055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:20.450618 25068 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:11:20.452961 25068 webserver.cc:460] Webserver started at http://127.0.0.1:33521/ using document root <none> and password file <none>
I20260924 16:11:20.453989 25068 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:20.454185 25068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:20.454578 25068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:20.458709 25068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "ac1530964b984c7baf18ce78b17c223e"
format_stamp: "Formatted at 2026-09-24 16:11:20 on dist-test-slave-1gc3"
server_key: "d5fb1980081867f1664c84c0a619d1cc"
server_key_iv: "6da266a13b9843f99348b66b740b0ff3"
server_key_version: "encryptionkey@0"
I20260924 16:11:20.459743 25068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "ac1530964b984c7baf18ce78b17c223e"
format_stamp: "Formatted at 2026-09-24 16:11:20 on dist-test-slave-1gc3"
server_key: "d5fb1980081867f1664c84c0a619d1cc"
server_key_iv: "6da266a13b9843f99348b66b740b0ff3"
server_key_version: "encryptionkey@0"
I20260924 16:11:20.467221 25068 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260924 16:11:20.472126 25083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:20.474181 25068 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260924 16:11:20.474453 25068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "ac1530964b984c7baf18ce78b17c223e"
format_stamp: "Formatted at 2026-09-24 16:11:20 on dist-test-slave-1gc3"
server_key: "d5fb1980081867f1664c84c0a619d1cc"
server_key_iv: "6da266a13b9843f99348b66b740b0ff3"
server_key_version: "encryptionkey@0"
I20260924 16:11:20.474706 25068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:20.495471 25068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:20.501258 25068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:20.501646 25068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:20.524515 25068 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44913
I20260924 16:11:20.524555 25134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44913 every 8 connection(s)
I20260924 16:11:20.526467 25068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:20.527976 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 25068
I20260924 16:11:20.528199 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:11:20.528820 3931 external_mini_cluster.cc:1428] Setting key ffd133aa22324ddb4c66aeea8c33fbe6
I20260924 16:11:20.531957 25135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:20.546079 25135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac1530964b984c7baf18ce78b17c223e: Bootstrap starting.
I20260924 16:11:20.546837 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:20.531243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57883 (local address 127.0.0.1:44913)
0924 16:11:20.532260 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:20.532281 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:20.532341 (+ 60us) server_negotiation.cc:407] Connection header received
0924 16:11:20.534453 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:20.534489 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:20.535126 (+ 637us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:20.535610 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:20.536419 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:20.538324 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:20.540391 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:20.541437 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:20.541647 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:20.543257 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:20.543269 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:20.543299 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:20.544113 (+ 814us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:20.544231 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:20.544265 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:20.544399 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:20.544731 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 16:11:20.545164 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":426,"threads_started":1}
I20260924 16:11:20.550200 25135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac1530964b984c7baf18ce78b17c223e: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:20.551489 25135 log.cc:824] T 00000000000000000000000000000000 P ac1530964b984c7baf18ce78b17c223e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:20.552392 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:11:20.555099 25135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac1530964b984c7baf18ce78b17c223e: No bootstrap required, opened a new log
I20260924 16:11:20.563840 25135 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac1530964b984c7baf18ce78b17c223e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1530964b984c7baf18ce78b17c223e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44913 } }
I20260924 16:11:20.564360 25135 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac1530964b984c7baf18ce78b17c223e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:20.564545 25135 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac1530964b984c7baf18ce78b17c223e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1530964b984c7baf18ce78b17c223e, State: Initialized, Role: FOLLOWER
I20260924 16:11:20.565094 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:20.557868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50879 (local address 127.0.0.1:44913)
0924 16:11:20.558058 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:20.558064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:20.558647 (+ 583us) server_negotiation.cc:407] Connection header received
0924 16:11:20.558895 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:20.558899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:20.558986 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:20.559105 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:20.559665 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:20.560555 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:20.562046 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:20.562668 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:20.562807 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:20.564219 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:20.564224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:20.564228 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:20.564254 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:20.564313 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:20.564336 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:20.564437 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:20.564598 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 16:11:20.564719 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:11:20.565485 25135 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac1530964b984c7baf18ce78b17c223e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1530964b984c7baf18ce78b17c223e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44913 } }
I20260924 16:11:20.565793 25135 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac1530964b984c7baf18ce78b17c223e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:20.565953 25135 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac1530964b984c7baf18ce78b17c223e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:20.566210 25135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac1530964b984c7baf18ce78b17c223e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:20.568672 25135 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac1530964b984c7baf18ce78b17c223e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1530964b984c7baf18ce78b17c223e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44913 } }
I20260924 16:11:20.569280 25135 leader_election.cc:302] T 00000000000000000000000000000000 P ac1530964b984c7baf18ce78b17c223e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac1530964b984c7baf18ce78b17c223e; no voters:
I20260924 16:11:20.569979 25135 leader_election.cc:288] T 00000000000000000000000000000000 P ac1530964b984c7baf18ce78b17c223e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:20.570232 25145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac1530964b984c7baf18ce78b17c223e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:20.571213 25145 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac1530964b984c7baf18ce78b17c223e [term 1 LEADER]: Becoming Leader. State: Replica: ac1530964b984c7baf18ce78b17c223e, State: Running, Role: LEADER
I20260924 16:11:20.572010 25145 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac1530964b984c7baf18ce78b17c223e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1530964b984c7baf18ce78b17c223e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44913 } }
I20260924 16:11:20.572605 25135 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac1530964b984c7baf18ce78b17c223e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:20.576634 25146 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac1530964b984c7baf18ce78b17c223e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac1530964b984c7baf18ce78b17c223e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1530964b984c7baf18ce78b17c223e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44913 } } }
I20260924 16:11:20.577176 25146 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac1530964b984c7baf18ce78b17c223e [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:20.577884 25147 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac1530964b984c7baf18ce78b17c223e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac1530964b984c7baf18ce78b17c223e. Latest consensus state: current_term: 1 leader_uuid: "ac1530964b984c7baf18ce78b17c223e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1530964b984c7baf18ce78b17c223e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44913 } } }
I20260924 16:11:20.578454 25147 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac1530964b984c7baf18ce78b17c223e [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:20.579916 25154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:20.585544 25154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:20.603883 25154 catalog_manager.cc:1347] Generated new cluster ID: b98ec8a6669a46ad960d11add7a804cb
I20260924 16:11:20.604064 25154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:20.640304 25154 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:20.641985 25154 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:20.652136 25154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac1530964b984c7baf18ce78b17c223e: Generated new TSK 0
I20260924 16:11:20.653363 25154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:11:20.670588 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 25068
W20260924 16:11:20.672178 25130 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57883
W20260924 16:11:20.673193 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 25068: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:11:20.673369 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 25068
2026-09-24T16:11:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2101 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:20 dist-test-slave-1gc3 krb5kdc[25166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:20 dist-test-slave-1gc3 krb5kdc[25166](info): set up 2 sockets
Sep 24 16:11:20 dist-test-slave-1gc3 krb5kdc[25166](info): commencing operation
krb5kdc: starting...
W20260924 16:11:21.755858 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:11:21 dist-test-slave-1gc3 krb5kdc[25166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:11:22.086966 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41967
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:22.510347 25179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:22.510910 25179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:22.511148 25179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:22.519846 25179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:22.520043 25179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:22.520128 25179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:22.520236 25179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:22.520310 25179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:22.532325 25179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:22.534245 25179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:22.536022 25179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:22.550746 25184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:22.550873 25185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:22.551628 25187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:22.552570 25179 server_base.cc:1034] running on GCE node
I20260924 16:11:22.553632 25179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:22.555428 25179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:22.556735 25179 hybrid_clock.cc:648] HybridClock initialized: now 1790266282556642 us; error 66 us; skew 500 ppm
I20260924 16:11:22.557231 25179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:22 dist-test-slave-1gc3 krb5kdc[25166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:22.561120 25179 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:11:22.563248 25179 webserver.cc:460] Webserver started at http://127.0.0.1:43777/ using document root <none> and password file <none>
I20260924 16:11:22.564133 25179 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:22.564265 25179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:22.564769 25179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:22.568773 25179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "e2647a6d1b1141989425842e5bc1559a"
format_stamp: "Formatted at 2026-09-24 16:11:22 on dist-test-slave-1gc3"
server_key: "af878cfdda3fc54fffaa909e7f28fc7d"
server_key_iv: "dbaea98d043f0d36bed76d203bb38c50"
server_key_version: "encryptionkey@0"
I20260924 16:11:22.569777 25179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "e2647a6d1b1141989425842e5bc1559a"
format_stamp: "Formatted at 2026-09-24 16:11:22 on dist-test-slave-1gc3"
server_key: "af878cfdda3fc54fffaa909e7f28fc7d"
server_key_iv: "dbaea98d043f0d36bed76d203bb38c50"
server_key_version: "encryptionkey@0"
I20260924 16:11:22.577052 25179 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 16:11:22.582086 25194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:22.584277 25179 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260924 16:11:22.584532 25179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "e2647a6d1b1141989425842e5bc1559a"
format_stamp: "Formatted at 2026-09-24 16:11:22 on dist-test-slave-1gc3"
server_key: "af878cfdda3fc54fffaa909e7f28fc7d"
server_key_iv: "dbaea98d043f0d36bed76d203bb38c50"
server_key_version: "encryptionkey@0"
I20260924 16:11:22.584770 25179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:22.592049 25179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:22.598431 25179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:22.598809 25179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:22.620513 25179 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41967
I20260924 16:11:22.620594 25245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41967 every 8 connection(s)
I20260924 16:11:22.622207 25179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:22.622942 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 25179
I20260924 16:11:22.623160 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:11:22.623893 3931 external_mini_cluster.cc:1428] Setting key 85ada6d7f015ef65d580bab45502d657
I20260924 16:11:22.627578 25246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:22.634200 25249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:22.626276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40215 (local address 127.0.0.1:41967)
0924 16:11:22.627375 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:22.627392 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:22.627454 (+ 62us) server_negotiation.cc:407] Connection header received
0924 16:11:22.629513 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:22.629547 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:22.630055 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:22.630577 (+ 522us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:22.631870 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:22.631879 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:22.631897 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:22.631920 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:22.632040 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:22.632065 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:22.632300 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:22.632426 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 16:11:22.632723 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":431,"threads_started":1}
I20260924 16:11:22.639624 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:11:22.641891 25246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2647a6d1b1141989425842e5bc1559a: Bootstrap starting.
I20260924 16:11:22.646346 25246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2647a6d1b1141989425842e5bc1559a: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:22.647663 25246 log.cc:824] T 00000000000000000000000000000000 P e2647a6d1b1141989425842e5bc1559a: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:22.649281 25249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:22.645957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57487 (local address 127.0.0.1:41967)
0924 16:11:22.646284 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:22.646295 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:22.646903 (+ 608us) server_negotiation.cc:407] Connection header received
0924 16:11:22.647089 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:22.647095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:22.647173 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:22.647331 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:22.648467 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:22.648471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:22.648474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:22.648495 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:22.648542 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:22.648558 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:22.648668 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:22.648818 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 16:11:22.648876 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 16:11:22.652061 25246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2647a6d1b1141989425842e5bc1559a: No bootstrap required, opened a new log
I20260924 16:11:22.661398 25246 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2647a6d1b1141989425842e5bc1559a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2647a6d1b1141989425842e5bc1559a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41967 } }
I20260924 16:11:22.661931 25246 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2647a6d1b1141989425842e5bc1559a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:22.662039 25246 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2647a6d1b1141989425842e5bc1559a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2647a6d1b1141989425842e5bc1559a, State: Initialized, Role: FOLLOWER
I20260924 16:11:22.662833 25246 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2647a6d1b1141989425842e5bc1559a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2647a6d1b1141989425842e5bc1559a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41967 } }
I20260924 16:11:22.663106 25246 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2647a6d1b1141989425842e5bc1559a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:22.663271 25246 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2647a6d1b1141989425842e5bc1559a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:22.663448 25246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2647a6d1b1141989425842e5bc1559a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:22.666045 25246 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2647a6d1b1141989425842e5bc1559a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2647a6d1b1141989425842e5bc1559a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41967 } }
I20260924 16:11:22.666766 25246 leader_election.cc:302] T 00000000000000000000000000000000 P e2647a6d1b1141989425842e5bc1559a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2647a6d1b1141989425842e5bc1559a; no voters:
I20260924 16:11:22.667622 25246 leader_election.cc:288] T 00000000000000000000000000000000 P e2647a6d1b1141989425842e5bc1559a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:22.667819 25256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2647a6d1b1141989425842e5bc1559a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:22.668632 25256 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2647a6d1b1141989425842e5bc1559a [term 1 LEADER]: Becoming Leader. State: Replica: e2647a6d1b1141989425842e5bc1559a, State: Running, Role: LEADER
I20260924 16:11:22.669286 25256 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2647a6d1b1141989425842e5bc1559a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2647a6d1b1141989425842e5bc1559a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41967 } }
I20260924 16:11:22.670187 25246 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2647a6d1b1141989425842e5bc1559a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:22.673072 25257 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2647a6d1b1141989425842e5bc1559a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2647a6d1b1141989425842e5bc1559a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2647a6d1b1141989425842e5bc1559a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41967 } } }
I20260924 16:11:22.673735 25257 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2647a6d1b1141989425842e5bc1559a [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:22.673031 25258 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2647a6d1b1141989425842e5bc1559a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2647a6d1b1141989425842e5bc1559a. Latest consensus state: current_term: 1 leader_uuid: "e2647a6d1b1141989425842e5bc1559a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2647a6d1b1141989425842e5bc1559a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41967 } } }
I20260924 16:11:22.674767 25258 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2647a6d1b1141989425842e5bc1559a [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:22.676425 25265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:22.682422 25265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:22.695613 25265 catalog_manager.cc:1347] Generated new cluster ID: 9d92d3e18bdf4e85b5a991efb037980b
I20260924 16:11:22.695783 25265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:22.710457 25265 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:22.711167 25265 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:22.734444 25265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2647a6d1b1141989425842e5bc1559a: Generated new TSK 0
I20260924 16:11:22.735685 25265 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:11:22.752277 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 25179
W20260924 16:11:22.754334 25241 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40215
W20260924 16:11:22.755201 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 25179: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:11:22.755404 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 25179
2026-09-24T16:11:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2081 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:22 dist-test-slave-1gc3 krb5kdc[25277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:22 dist-test-slave-1gc3 krb5kdc[25277](info): set up 2 sockets
Sep 24 16:11:22 dist-test-slave-1gc3 krb5kdc[25277](info): commencing operation
krb5kdc: starting...
W20260924 16:11:23.836403 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:11:24 dist-test-slave-1gc3 krb5kdc[25277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:11:24.165969 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:24.592800 25290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:24.593369 25290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:24.593544 25290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:24.603042 25290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:24.603210 25290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:24.603273 25290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:24.603354 25290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:24.603404 25290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:24.614979 25290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:24.616813 25290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:24.618556 25290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:24.632709 25295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:24.632824 25296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:24.632807 25298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:24.633963 25290 server_base.cc:1034] running on GCE node
I20260924 16:11:24.634959 25290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:24.636642 25290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:24.637926 25290 hybrid_clock.cc:648] HybridClock initialized: now 1790266284637853 us; error 74 us; skew 500 ppm
I20260924 16:11:24.638387 25290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:24 dist-test-slave-1gc3 krb5kdc[25277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:24.641932 25290 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:11:24.643792 25290 webserver.cc:460] Webserver started at http://127.0.0.1:40351/ using document root <none> and password file <none>
I20260924 16:11:24.644595 25290 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:24.644719 25290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:24.645073 25290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:24.648370 25290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "2f58070be5ba48ffa99aeabc0fbbcadb"
format_stamp: "Formatted at 2026-09-24 16:11:24 on dist-test-slave-1gc3"
server_key: "2f3972dbb5f3b851208e8e4980bbdb38"
server_key_iv: "bd33e56ffe4a28856c12c3241a204d56"
server_key_version: "encryptionkey@0"
I20260924 16:11:24.649170 25290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "2f58070be5ba48ffa99aeabc0fbbcadb"
format_stamp: "Formatted at 2026-09-24 16:11:24 on dist-test-slave-1gc3"
server_key: "2f3972dbb5f3b851208e8e4980bbdb38"
server_key_iv: "bd33e56ffe4a28856c12c3241a204d56"
server_key_version: "encryptionkey@0"
I20260924 16:11:24.655699 25290 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260924 16:11:24.660143 25305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:24.661886 25290 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:11:24.662151 25290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "2f58070be5ba48ffa99aeabc0fbbcadb"
format_stamp: "Formatted at 2026-09-24 16:11:24 on dist-test-slave-1gc3"
server_key: "2f3972dbb5f3b851208e8e4980bbdb38"
server_key_iv: "bd33e56ffe4a28856c12c3241a204d56"
server_key_version: "encryptionkey@0"
I20260924 16:11:24.662370 25290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:24.680650 25290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:24.686770 25290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:24.687101 25290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:24.709889 25290 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37831
I20260924 16:11:24.709955 25356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37831 every 8 connection(s)
I20260924 16:11:24.711504 25290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:24.716810 25357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:24.721876 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 25290
I20260924 16:11:24.722152 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:11:24.722790 3931 external_mini_cluster.cc:1428] Setting key 051358f19fd9927b0aa4a463aa91f112
I20260924 16:11:24.735173 25357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f58070be5ba48ffa99aeabc0fbbcadb: Bootstrap starting.
I20260924 16:11:24.740060 25357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f58070be5ba48ffa99aeabc0fbbcadb: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:24.741849 25357 log.cc:824] T 00000000000000000000000000000000 P 2f58070be5ba48ffa99aeabc0fbbcadb: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:11:24 dist-test-slave-1gc3 krb5kdc[25277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 16:11:24.746192 25357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f58070be5ba48ffa99aeabc0fbbcadb: No bootstrap required, opened a new log
I20260924 16:11:24.753059 25360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:24.725490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:40949 (local address 127.0.0.1:37831)
0924 16:11:24.726818 (+ 1328us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:24.726844 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:24.726928 (+ 84us) server_negotiation.cc:407] Connection header received
0924 16:11:24.729916 (+ 2988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:24.729959 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:24.730630 (+ 671us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:24.731059 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:24.733230 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:24.734885 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:24.736952 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:24.737858 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:24.738043 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:24.744262 (+ 6219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:24.744273 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:24.744295 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:24.744342 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:24.747840 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:24.749060 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:24.749071 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:24.749080 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:24.749306 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:24.749791 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:24.749795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:24.749797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:24.750557 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:24.750923 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:24.751439 (+ 516us) server_negotiation.cc:299] Negotiation successful
0924 16:11:24.751844 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":913,"thread_start_us":543,"threads_started":1}
I20260924 16:11:24.757395 25357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f58070be5ba48ffa99aeabc0fbbcadb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f58070be5ba48ffa99aeabc0fbbcadb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37831 } }
I20260924 16:11:24.757915 25357 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f58070be5ba48ffa99aeabc0fbbcadb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:24.758023 25357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f58070be5ba48ffa99aeabc0fbbcadb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f58070be5ba48ffa99aeabc0fbbcadb, State: Initialized, Role: FOLLOWER
I20260924 16:11:24.758255 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:11:24.758879 25357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f58070be5ba48ffa99aeabc0fbbcadb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f58070be5ba48ffa99aeabc0fbbcadb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37831 } }
I20260924 16:11:24.759204 25357 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f58070be5ba48ffa99aeabc0fbbcadb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:24.759362 25357 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f58070be5ba48ffa99aeabc0fbbcadb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:24.759545 25357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f58070be5ba48ffa99aeabc0fbbcadb [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:24.763266 25357 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f58070be5ba48ffa99aeabc0fbbcadb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f58070be5ba48ffa99aeabc0fbbcadb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37831 } }
I20260924 16:11:24.764060 25357 leader_election.cc:302] T 00000000000000000000000000000000 P 2f58070be5ba48ffa99aeabc0fbbcadb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f58070be5ba48ffa99aeabc0fbbcadb; no voters:
I20260924 16:11:24.764883 25357 leader_election.cc:288] T 00000000000000000000000000000000 P 2f58070be5ba48ffa99aeabc0fbbcadb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:24.765066 25367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f58070be5ba48ffa99aeabc0fbbcadb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:24.769147 25367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f58070be5ba48ffa99aeabc0fbbcadb [term 1 LEADER]: Becoming Leader. State: Replica: 2f58070be5ba48ffa99aeabc0fbbcadb, State: Running, Role: LEADER
I20260924 16:11:24.769991 25367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f58070be5ba48ffa99aeabc0fbbcadb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f58070be5ba48ffa99aeabc0fbbcadb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37831 } }
I20260924 16:11:24.774022 25369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f58070be5ba48ffa99aeabc0fbbcadb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f58070be5ba48ffa99aeabc0fbbcadb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f58070be5ba48ffa99aeabc0fbbcadb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37831 } } }
I20260924 16:11:24.774554 25369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f58070be5ba48ffa99aeabc0fbbcadb [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:24.774786 25357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f58070be5ba48ffa99aeabc0fbbcadb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:24.775759 25368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f58070be5ba48ffa99aeabc0fbbcadb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f58070be5ba48ffa99aeabc0fbbcadb. Latest consensus state: current_term: 1 leader_uuid: "2f58070be5ba48ffa99aeabc0fbbcadb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f58070be5ba48ffa99aeabc0fbbcadb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37831 } } }
I20260924 16:11:24.777714 25368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f58070be5ba48ffa99aeabc0fbbcadb [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:24.778707 25371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:24.781316 25360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:24.764303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:49029 (local address 127.0.0.1:37831)
0924 16:11:24.764533 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:24.764541 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:24.764950 (+ 409us) server_negotiation.cc:407] Connection header received
0924 16:11:24.765142 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:24.765146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:24.765234 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:24.765339 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:24.767319 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:24.768153 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:24.769422 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:24.769897 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:24.769977 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:24.772992 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:24.772999 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:24.773004 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:24.773050 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:24.777024 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:24.777968 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:24.777974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:24.777978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:24.778160 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:24.778978 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:24.778982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:24.778984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:24.779523 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:24.779913 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:24.780618 (+ 705us) server_negotiation.cc:299] Negotiation successful
0924 16:11:24.780866 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:11:24.786353 25371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:24.804231 25371 catalog_manager.cc:1347] Generated new cluster ID: 2e8ead3c41044d36bc7c5b2fff9aaa5c
I20260924 16:11:24.804430 25371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:24.853652 25371 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:24.855386 25371 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:24.863162 25371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f58070be5ba48ffa99aeabc0fbbcadb: Generated new TSK 0
I20260924 16:11:24.864421 25371 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:11:24.885895 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 25290
I20260924 16:11:24.887776 25352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:24.998662 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 25290
W20260924 16:11:25.016572 25275 connection.cc:535] client connection to 127.0.0.1:37831 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37831): Connection reset by peer (error 104)
2026-09-24T16:11:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2242 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:25 dist-test-slave-1gc3 krb5kdc[25389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:25 dist-test-slave-1gc3 krb5kdc[25389](info): set up 2 sockets
Sep 24 16:11:25 dist-test-slave-1gc3 krb5kdc[25389](info): commencing operation
krb5kdc: starting...
W20260924 16:11:26.068420 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.005s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:11:26 dist-test-slave-1gc3 krb5kdc[25389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:11:26.394361 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36311
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:26.784457 25402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:26.784814 25402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:26.784976 25402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:26.792814 25402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:26.792979 25402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:26.793066 25402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:26.793149 25402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:26.793212 25402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:26.804430 25402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:26.805964 25402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:26.807482 25402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:26.820880 25407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:26.820962 25410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:26.820951 25408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:26.822451 25402 server_base.cc:1034] running on GCE node
I20260924 16:11:26.823388 25402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:26.824940 25402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:26.826239 25402 hybrid_clock.cc:648] HybridClock initialized: now 1790266286826163 us; error 68 us; skew 500 ppm
I20260924 16:11:26.826643 25402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:26 dist-test-slave-1gc3 krb5kdc[25389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:26.830379 25402 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:11:26.832223 25402 webserver.cc:460] Webserver started at http://127.0.0.1:40815/ using document root <none> and password file <none>
I20260924 16:11:26.832963 25402 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:26.833089 25402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:26.833405 25402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:26.836687 25402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "7d1cd788064b4285aa126adc6afcbb6f"
format_stamp: "Formatted at 2026-09-24 16:11:26 on dist-test-slave-1gc3"
server_key: "c3f13e22381a5ac639234daebde07235"
server_key_iv: "839ba7b26defc58210ac15e6679e51c3"
server_key_version: "encryptionkey@0"
I20260924 16:11:26.837519 25402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "7d1cd788064b4285aa126adc6afcbb6f"
format_stamp: "Formatted at 2026-09-24 16:11:26 on dist-test-slave-1gc3"
server_key: "c3f13e22381a5ac639234daebde07235"
server_key_iv: "839ba7b26defc58210ac15e6679e51c3"
server_key_version: "encryptionkey@0"
I20260924 16:11:26.843880 25402 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260924 16:11:26.848827 25417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:26.850874 25402 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:11:26.851119 25402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "7d1cd788064b4285aa126adc6afcbb6f"
format_stamp: "Formatted at 2026-09-24 16:11:26 on dist-test-slave-1gc3"
server_key: "c3f13e22381a5ac639234daebde07235"
server_key_iv: "839ba7b26defc58210ac15e6679e51c3"
server_key_version: "encryptionkey@0"
I20260924 16:11:26.851349 25402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:26.865701 25402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:26.871274 25402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:26.871574 25402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:26.892323 25402 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36311
I20260924 16:11:26.892411 25468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36311 every 8 connection(s)
I20260924 16:11:26.893908 25402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:26.899013 25469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:26.899904 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 25402
I20260924 16:11:26.900121 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:11:26.900784 3931 external_mini_cluster.cc:1428] Setting key e9db1408123070ec1309678497ca581f
I20260924 16:11:26.916632 25469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d1cd788064b4285aa126adc6afcbb6f: Bootstrap starting.
I20260924 16:11:26.917817 25472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:26.903152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:34009 (local address 127.0.0.1:36311)
0924 16:11:26.904074 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:26.904087 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:26.904142 (+ 55us) server_negotiation.cc:407] Connection header received
0924 16:11:26.906789 (+ 2647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:26.906821 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:26.907379 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:26.907783 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:26.908640 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:26.910078 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:26.912226 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:26.913170 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:26.913349 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:26.915084 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:26.915093 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:26.915113 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:26.915748 (+ 635us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:26.915851 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:26.915882 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:26.916073 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:26.916380 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 16:11:26.916613 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":470,"threads_started":1}
I20260924 16:11:26.921384 25469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d1cd788064b4285aa126adc6afcbb6f: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:26.922883 25469 log.cc:824] T 00000000000000000000000000000000 P 7d1cd788064b4285aa126adc6afcbb6f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:26.923225 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:11:26.926791 25469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d1cd788064b4285aa126adc6afcbb6f: No bootstrap required, opened a new log
I20260924 16:11:26.934973 25472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:26.928507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:46129 (local address 127.0.0.1:36311)
0924 16:11:26.928698 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:26.928704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:26.929137 (+ 433us) server_negotiation.cc:407] Connection header received
0924 16:11:26.929347 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:26.929352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:26.929422 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:26.929514 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:26.930141 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:26.930965 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:26.932319 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:26.932813 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:26.932949 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:26.934179 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:26.934185 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:26.934189 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:26.934211 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:26.934283 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:26.934299 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:26.934422 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:26.934600 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 16:11:26.934666 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 16:11:26.936411 25469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d1cd788064b4285aa126adc6afcbb6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1cd788064b4285aa126adc6afcbb6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36311 } }
I20260924 16:11:26.937027 25469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d1cd788064b4285aa126adc6afcbb6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:26.937139 25469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d1cd788064b4285aa126adc6afcbb6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1cd788064b4285aa126adc6afcbb6f, State: Initialized, Role: FOLLOWER
I20260924 16:11:26.937907 25469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d1cd788064b4285aa126adc6afcbb6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1cd788064b4285aa126adc6afcbb6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36311 } }
I20260924 16:11:26.938205 25469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d1cd788064b4285aa126adc6afcbb6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:26.938373 25469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d1cd788064b4285aa126adc6afcbb6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:26.938560 25469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d1cd788064b4285aa126adc6afcbb6f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:26.941414 25469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d1cd788064b4285aa126adc6afcbb6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1cd788064b4285aa126adc6afcbb6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36311 } }
I20260924 16:11:26.942045 25469 leader_election.cc:302] T 00000000000000000000000000000000 P 7d1cd788064b4285aa126adc6afcbb6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d1cd788064b4285aa126adc6afcbb6f; no voters:
I20260924 16:11:26.942957 25469 leader_election.cc:288] T 00000000000000000000000000000000 P 7d1cd788064b4285aa126adc6afcbb6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:26.943157 25479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d1cd788064b4285aa126adc6afcbb6f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:26.944231 25479 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d1cd788064b4285aa126adc6afcbb6f [term 1 LEADER]: Becoming Leader. State: Replica: 7d1cd788064b4285aa126adc6afcbb6f, State: Running, Role: LEADER
I20260924 16:11:26.944942 25479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d1cd788064b4285aa126adc6afcbb6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1cd788064b4285aa126adc6afcbb6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36311 } }
I20260924 16:11:26.945708 25469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d1cd788064b4285aa126adc6afcbb6f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:26.947932 25481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d1cd788064b4285aa126adc6afcbb6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d1cd788064b4285aa126adc6afcbb6f. Latest consensus state: current_term: 1 leader_uuid: "7d1cd788064b4285aa126adc6afcbb6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1cd788064b4285aa126adc6afcbb6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36311 } } }
I20260924 16:11:26.949019 25481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d1cd788064b4285aa126adc6afcbb6f [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:26.948493 25480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d1cd788064b4285aa126adc6afcbb6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d1cd788064b4285aa126adc6afcbb6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1cd788064b4285aa126adc6afcbb6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36311 } } }
I20260924 16:11:26.949227 25480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d1cd788064b4285aa126adc6afcbb6f [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:26.951187 25484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:26.957438 25484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:26.974364 25484 catalog_manager.cc:1347] Generated new cluster ID: c0a5cafe6fcc484a8fe869bea735f178
I20260924 16:11:26.974555 25484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:26.986791 25484 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:26.989050 25484 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:26.998658 25484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d1cd788064b4285aa126adc6afcbb6f: Generated new TSK 0
I20260924 16:11:26.999738 25484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:11:27.017549 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 25402
W20260924 16:11:27.019018 25464 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.27:34009
W20260924 16:11:27.019811 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 25402: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:11:27.020025 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 25402
W20260924 16:11:27.037238 25387 connection.cc:535] client connection to 127.0.0.1:36311 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36311): Connection reset by peer (error 104)
2026-09-24T16:11:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2019 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:27 dist-test-slave-1gc3 krb5kdc[25500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:27 dist-test-slave-1gc3 krb5kdc[25500](info): set up 2 sockets
Sep 24 16:11:27 dist-test-slave-1gc3 krb5kdc[25500](info): commencing operation
krb5kdc: starting...
W20260924 16:11:28.096917 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:11:28 dist-test-slave-1gc3 krb5kdc[25500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:11:28.442497 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42507
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:28.806056 25513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:28.806439 25513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:28.806538 25513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:28.814272 25513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:28.814409 25513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:28.814468 25513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:28.814534 25513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:28.814584 25513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:28.825011 25513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:28.826409 25513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:28.827724 25513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:28.841286 25518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:28.841342 25519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:28.841367 25521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:28.842604 25513 server_base.cc:1034] running on GCE node
I20260924 16:11:28.843760 25513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:28.845913 25513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:28.847205 25513 hybrid_clock.cc:648] HybridClock initialized: now 1790266288847108 us; error 49 us; skew 500 ppm
I20260924 16:11:28.847643 25513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:28 dist-test-slave-1gc3 krb5kdc[25500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:28.856629 25513 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:11:28.858911 25513 webserver.cc:460] Webserver started at http://127.0.0.1:44949/ using document root <none> and password file <none>
I20260924 16:11:28.859894 25513 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:28.860066 25513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:28.860459 25513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:28.864620 25513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "c67a6f3768974c09a43652547a763f45"
format_stamp: "Formatted at 2026-09-24 16:11:28 on dist-test-slave-1gc3"
server_key: "f64ddb65a809e572f7366861f863d377"
server_key_iv: "b9c533b1c83cc27b74e7d77e0efd5b87"
server_key_version: "encryptionkey@0"
I20260924 16:11:28.865742 25513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "c67a6f3768974c09a43652547a763f45"
format_stamp: "Formatted at 2026-09-24 16:11:28 on dist-test-slave-1gc3"
server_key: "f64ddb65a809e572f7366861f863d377"
server_key_iv: "b9c533b1c83cc27b74e7d77e0efd5b87"
server_key_version: "encryptionkey@0"
I20260924 16:11:28.873029 25513 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260924 16:11:28.877758 25528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:28.879673 25513 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:11:28.879887 25513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "c67a6f3768974c09a43652547a763f45"
format_stamp: "Formatted at 2026-09-24 16:11:28 on dist-test-slave-1gc3"
server_key: "f64ddb65a809e572f7366861f863d377"
server_key_iv: "b9c533b1c83cc27b74e7d77e0efd5b87"
server_key_version: "encryptionkey@0"
I20260924 16:11:28.880137 25513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:28.887145 25513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:28.891894 25513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:28.892218 25513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:28.914933 25513 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42507
I20260924 16:11:28.915036 25579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42507 every 8 connection(s)
I20260924 16:11:28.916719 25513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:28.921326 25580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:28.926625 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 25513
I20260924 16:11:28.926831 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:11:28.927376 3931 external_mini_cluster.cc:1428] Setting key dc67f14f8223cf58dd1c424bd249f95d
I20260924 16:11:28.935238 25580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c67a6f3768974c09a43652547a763f45: Bootstrap starting.
I20260924 16:11:28.938012 25583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:28.929850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:39471 (local address 127.0.0.1:42507)
0924 16:11:28.930927 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:28.930943 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:28.931002 (+ 59us) server_negotiation.cc:407] Connection header received
0924 16:11:28.933370 (+ 2368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:28.933411 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:28.933980 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:28.934513 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:28.935884 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:28.935894 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:28.935914 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:28.935932 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:28.936021 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:28.936046 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:28.936277 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:28.936400 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 16:11:28.936643 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":476,"threads_started":1}
I20260924 16:11:28.939833 25580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c67a6f3768974c09a43652547a763f45: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:28.941107 25580 log.cc:824] T 00000000000000000000000000000000 P c67a6f3768974c09a43652547a763f45: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:28.943588 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:11:28.944871 25580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c67a6f3768974c09a43652547a763f45: No bootstrap required, opened a new log
I20260924 16:11:28.952230 25583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:28.949184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:43997 (local address 127.0.0.1:42507)
0924 16:11:28.949456 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:28.949462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:28.949862 (+ 400us) server_negotiation.cc:407] Connection header received
0924 16:11:28.950094 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:28.950134 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:28.950215 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:28.950340 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:28.951426 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:28.951431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:28.951434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:28.951451 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:28.951512 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:28.951529 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:28.951653 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:28.951785 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 16:11:28.951850 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260924 16:11:28.954501 25580 raft_consensus.cc:348] T 00000000000000000000000000000000 P c67a6f3768974c09a43652547a763f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67a6f3768974c09a43652547a763f45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42507 } }
I20260924 16:11:28.954988 25580 raft_consensus.cc:374] T 00000000000000000000000000000000 P c67a6f3768974c09a43652547a763f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:28.955088 25580 raft_consensus.cc:729] T 00000000000000000000000000000000 P c67a6f3768974c09a43652547a763f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67a6f3768974c09a43652547a763f45, State: Initialized, Role: FOLLOWER
I20260924 16:11:28.955767 25580 consensus_queue.cc:260] T 00000000000000000000000000000000 P c67a6f3768974c09a43652547a763f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67a6f3768974c09a43652547a763f45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42507 } }
I20260924 16:11:28.956039 25580 raft_consensus.cc:388] T 00000000000000000000000000000000 P c67a6f3768974c09a43652547a763f45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:28.956180 25580 raft_consensus.cc:482] T 00000000000000000000000000000000 P c67a6f3768974c09a43652547a763f45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:28.956440 25580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c67a6f3768974c09a43652547a763f45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:28.959331 25580 raft_consensus.cc:504] T 00000000000000000000000000000000 P c67a6f3768974c09a43652547a763f45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67a6f3768974c09a43652547a763f45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42507 } }
I20260924 16:11:28.959956 25580 leader_election.cc:302] T 00000000000000000000000000000000 P c67a6f3768974c09a43652547a763f45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c67a6f3768974c09a43652547a763f45; no voters:
I20260924 16:11:28.960642 25580 leader_election.cc:288] T 00000000000000000000000000000000 P c67a6f3768974c09a43652547a763f45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:28.960872 25590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c67a6f3768974c09a43652547a763f45 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:28.961835 25590 raft_consensus.cc:686] T 00000000000000000000000000000000 P c67a6f3768974c09a43652547a763f45 [term 1 LEADER]: Becoming Leader. State: Replica: c67a6f3768974c09a43652547a763f45, State: Running, Role: LEADER
I20260924 16:11:28.962638 25590 consensus_queue.cc:237] T 00000000000000000000000000000000 P c67a6f3768974c09a43652547a763f45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67a6f3768974c09a43652547a763f45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42507 } }
I20260924 16:11:28.963021 25580 sys_catalog.cc:564] T 00000000000000000000000000000000 P c67a6f3768974c09a43652547a763f45 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:28.966491 25592 sys_catalog.cc:455] T 00000000000000000000000000000000 P c67a6f3768974c09a43652547a763f45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c67a6f3768974c09a43652547a763f45. Latest consensus state: current_term: 1 leader_uuid: "c67a6f3768974c09a43652547a763f45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67a6f3768974c09a43652547a763f45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42507 } } }
I20260924 16:11:28.967451 25592 sys_catalog.cc:458] T 00000000000000000000000000000000 P c67a6f3768974c09a43652547a763f45 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:28.967041 25591 sys_catalog.cc:455] T 00000000000000000000000000000000 P c67a6f3768974c09a43652547a763f45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c67a6f3768974c09a43652547a763f45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67a6f3768974c09a43652547a763f45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42507 } } }
I20260924 16:11:28.968039 25591 sys_catalog.cc:458] T 00000000000000000000000000000000 P c67a6f3768974c09a43652547a763f45 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:28.968675 25599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:28.975662 25599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:28.987563 25599 catalog_manager.cc:1347] Generated new cluster ID: 1aa3296b33a344c6b3602ca580c2af6b
I20260924 16:11:28.987752 25599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:29.040191 25599 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:29.040874 25599 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:29.051699 25599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c67a6f3768974c09a43652547a763f45: Generated new TSK 0
I20260924 16:11:29.052947 25599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:11:29.066987 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 25513
W20260924 16:11:29.068578 25575 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.27:39471
W20260924 16:11:29.069653 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 25513: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:11:29.069864 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 25513
2026-09-24T16:11:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2049 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:29 dist-test-slave-1gc3 krb5kdc[25611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:29 dist-test-slave-1gc3 krb5kdc[25611](info): set up 2 sockets
Sep 24 16:11:29 dist-test-slave-1gc3 krb5kdc[25611](info): commencing operation
krb5kdc: starting...
W20260924 16:11:30.132766 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:11:30 dist-test-slave-1gc3 krb5kdc[25611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:11:30.458549 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36611
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:30.879928 25624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:30.880364 25624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:30.880517 25624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:30.888808 25624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:30.888999 25624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:30.889078 25624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:30.889169 25624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:30.889230 25624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:30.900924 25624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36611
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:30.903047 25624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:30.904829 25624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:30.919435 25632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:30.919520 25629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:30.920204 25630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:30.921077 25624 server_base.cc:1034] running on GCE node
I20260924 16:11:30.922355 25624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:30.924352 25624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:30.925609 25624 hybrid_clock.cc:648] HybridClock initialized: now 1790266290925546 us; error 52 us; skew 500 ppm
I20260924 16:11:30.926052 25624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:30 dist-test-slave-1gc3 krb5kdc[25611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:30.930063 25624 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:11:30.932654 25624 webserver.cc:460] Webserver started at http://127.0.0.1:45747/ using document root <none> and password file <none>
I20260924 16:11:30.933873 25624 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:30.934043 25624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:30.934473 25624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:30.938619 25624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "c56b241d71fd41c7a168c550864d758e"
format_stamp: "Formatted at 2026-09-24 16:11:30 on dist-test-slave-1gc3"
server_key: "86b92bb975f08907928c4962e8c375e9"
server_key_iv: "03fce1795995449e35bb611e543b4d13"
server_key_version: "encryptionkey@0"
I20260924 16:11:30.939639 25624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "c56b241d71fd41c7a168c550864d758e"
format_stamp: "Formatted at 2026-09-24 16:11:30 on dist-test-slave-1gc3"
server_key: "86b92bb975f08907928c4962e8c375e9"
server_key_iv: "03fce1795995449e35bb611e543b4d13"
server_key_version: "encryptionkey@0"
I20260924 16:11:30.946681 25624 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260924 16:11:30.951561 25639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:30.953754 25624 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260924 16:11:30.954030 25624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "c56b241d71fd41c7a168c550864d758e"
format_stamp: "Formatted at 2026-09-24 16:11:30 on dist-test-slave-1gc3"
server_key: "86b92bb975f08907928c4962e8c375e9"
server_key_iv: "03fce1795995449e35bb611e543b4d13"
server_key_version: "encryptionkey@0"
I20260924 16:11:30.954355 25624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:30.968947 25624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:30.973913 25624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:30.974285 25624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:30.997226 25624 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36611
I20260924 16:11:30.997272 25690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36611 every 8 connection(s)
I20260924 16:11:30.999090 25624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:31.003815 25691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:31.004382 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 25624
I20260924 16:11:31.004590 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:11:31.005121 3931 external_mini_cluster.cc:1428] Setting key ac9301935fdaa32db8a66348c2e95fc3
I20260924 16:11:31.020499 25691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c56b241d71fd41c7a168c550864d758e: Bootstrap starting.
Sep 24 16:11:31 dist-test-slave-1gc3 krb5kdc[25611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 16:11:31.025352 25691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c56b241d71fd41c7a168c550864d758e: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:31.026854 25691 log.cc:824] T 00000000000000000000000000000000 P c56b241d71fd41c7a168c550864d758e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:31.031468 25691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c56b241d71fd41c7a168c550864d758e: No bootstrap required, opened a new log
I20260924 16:11:31.036126 25694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:31.007602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:40767 (local address 127.0.0.1:36611)
0924 16:11:31.008655 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:31.008672 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:31.008733 (+ 61us) server_negotiation.cc:407] Connection header received
0924 16:11:31.010854 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:31.010890 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:31.011529 (+ 639us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:31.012018 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:31.014142 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:31.015943 (+ 1801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:31.018056 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:31.019170 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:31.019377 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:31.025516 (+ 6139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:31.025532 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:31.025568 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:31.025657 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:31.029556 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:31.031163 (+ 1607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:31.031177 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:31.031191 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:31.031503 (+ 312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:31.032233 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:31.032237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:31.032239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:31.033017 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:31.033432 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:31.034177 (+ 745us) server_negotiation.cc:299] Negotiation successful
0924 16:11:31.034626 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":737,"thread_start_us":444,"threads_started":1}
I20260924 16:11:31.041309 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:11:31.042137 25691 raft_consensus.cc:348] T 00000000000000000000000000000000 P c56b241d71fd41c7a168c550864d758e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56b241d71fd41c7a168c550864d758e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36611 } }
I20260924 16:11:31.042645 25691 raft_consensus.cc:374] T 00000000000000000000000000000000 P c56b241d71fd41c7a168c550864d758e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:31.042742 25691 raft_consensus.cc:729] T 00000000000000000000000000000000 P c56b241d71fd41c7a168c550864d758e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c56b241d71fd41c7a168c550864d758e, State: Initialized, Role: FOLLOWER
I20260924 16:11:31.043550 25691 consensus_queue.cc:260] T 00000000000000000000000000000000 P c56b241d71fd41c7a168c550864d758e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56b241d71fd41c7a168c550864d758e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36611 } }
I20260924 16:11:31.043850 25691 raft_consensus.cc:388] T 00000000000000000000000000000000 P c56b241d71fd41c7a168c550864d758e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:31.044008 25691 raft_consensus.cc:482] T 00000000000000000000000000000000 P c56b241d71fd41c7a168c550864d758e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:31.044202 25691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c56b241d71fd41c7a168c550864d758e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:31.047371 25691 raft_consensus.cc:504] T 00000000000000000000000000000000 P c56b241d71fd41c7a168c550864d758e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56b241d71fd41c7a168c550864d758e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36611 } }
I20260924 16:11:31.048460 25691 leader_election.cc:302] T 00000000000000000000000000000000 P c56b241d71fd41c7a168c550864d758e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c56b241d71fd41c7a168c550864d758e; no voters:
I20260924 16:11:31.049245 25691 leader_election.cc:288] T 00000000000000000000000000000000 P c56b241d71fd41c7a168c550864d758e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:31.049474 25701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c56b241d71fd41c7a168c550864d758e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:31.050901 25701 raft_consensus.cc:686] T 00000000000000000000000000000000 P c56b241d71fd41c7a168c550864d758e [term 1 LEADER]: Becoming Leader. State: Replica: c56b241d71fd41c7a168c550864d758e, State: Running, Role: LEADER
I20260924 16:11:31.051700 25701 consensus_queue.cc:237] T 00000000000000000000000000000000 P c56b241d71fd41c7a168c550864d758e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56b241d71fd41c7a168c550864d758e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36611 } }
I20260924 16:11:31.052132 25691 sys_catalog.cc:564] T 00000000000000000000000000000000 P c56b241d71fd41c7a168c550864d758e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:31.061606 25702 sys_catalog.cc:455] T 00000000000000000000000000000000 P c56b241d71fd41c7a168c550864d758e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c56b241d71fd41c7a168c550864d758e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56b241d71fd41c7a168c550864d758e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36611 } } }
I20260924 16:11:31.062201 25702 sys_catalog.cc:458] T 00000000000000000000000000000000 P c56b241d71fd41c7a168c550864d758e [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:31.067168 25703 sys_catalog.cc:455] T 00000000000000000000000000000000 P c56b241d71fd41c7a168c550864d758e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c56b241d71fd41c7a168c550864d758e. Latest consensus state: current_term: 1 leader_uuid: "c56b241d71fd41c7a168c550864d758e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56b241d71fd41c7a168c550864d758e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36611 } } }
I20260924 16:11:31.067621 25703 sys_catalog.cc:458] T 00000000000000000000000000000000 P c56b241d71fd41c7a168c550864d758e [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:31.069088 25694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:31.046946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:55897 (local address 127.0.0.1:36611)
0924 16:11:31.047183 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:31.047191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:31.047823 (+ 632us) server_negotiation.cc:407] Connection header received
0924 16:11:31.048132 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:31.048139 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:31.048265 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:31.048393 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:31.050592 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:31.052022 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:31.054012 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:31.054844 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:31.055063 (+ 219us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:31.059177 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:31.059187 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:31.059193 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:31.059290 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:31.064074 (+ 4784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:31.065422 (+ 1348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:31.065427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:31.065432 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:31.065658 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:31.066369 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:31.066378 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:31.066383 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:31.066929 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:31.068330 (+ 1401us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:31.068515 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 16:11:31.068699 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:11:31.069358 25708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:31.076980 25708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:31.090389 25708 catalog_manager.cc:1347] Generated new cluster ID: 39f70bb84fb54fdea36e11beb1fbbc5b
I20260924 16:11:31.090546 25708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:31.102556 25708 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:31.104276 25708 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:31.121248 25708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c56b241d71fd41c7a168c550864d758e: Generated new TSK 0
I20260924 16:11:31.122408 25708 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:11:31.130929 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 25624
I20260924 16:11:31.132828 25686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:31.238703 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 25624
W20260924 16:11:31.255178 25609 connection.cc:535] client connection to 127.0.0.1:36611 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36611): Connection reset by peer (error 104)
2026-09-24T16:11:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2168 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:31 dist-test-slave-1gc3 krb5kdc[25723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:31 dist-test-slave-1gc3 krb5kdc[25723](info): set up 2 sockets
Sep 24 16:11:31 dist-test-slave-1gc3 krb5kdc[25723](info): commencing operation
krb5kdc: starting...
W20260924 16:11:32.300253 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:11:32 dist-test-slave-1gc3 krb5kdc[25723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:11:32.655516 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41191
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:33.035965 25736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:33.036320 25736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:33.036425 25736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:33.045156 25736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:33.045315 25736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:33.045365 25736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:33.045437 25736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:33.045490 25736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:33.056977 25736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41191
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:33.058532 25736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:33.060029 25736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:33.073753 25741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:33.073844 25742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:33.073801 25744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:33.074894 25736 server_base.cc:1034] running on GCE node
I20260924 16:11:33.075865 25736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:33.077579 25736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:33.078858 25736 hybrid_clock.cc:648] HybridClock initialized: now 1790266293078780 us; error 60 us; skew 500 ppm
I20260924 16:11:33.079309 25736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:33 dist-test-slave-1gc3 krb5kdc[25723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:33.082785 25736 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:11:33.084576 25736 webserver.cc:460] Webserver started at http://127.0.0.1:33013/ using document root <none> and password file <none>
I20260924 16:11:33.085316 25736 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:33.085438 25736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:33.085758 25736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:33.089129 25736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "3c68916d0f504e4b9bfebc15cdbffd4b"
format_stamp: "Formatted at 2026-09-24 16:11:33 on dist-test-slave-1gc3"
server_key: "fc585d37dae1c0ebf0f13c75866f06ba"
server_key_iv: "5d21d6eff642af33c6f73606735d0858"
server_key_version: "encryptionkey@0"
I20260924 16:11:33.090039 25736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "3c68916d0f504e4b9bfebc15cdbffd4b"
format_stamp: "Formatted at 2026-09-24 16:11:33 on dist-test-slave-1gc3"
server_key: "fc585d37dae1c0ebf0f13c75866f06ba"
server_key_iv: "5d21d6eff642af33c6f73606735d0858"
server_key_version: "encryptionkey@0"
I20260924 16:11:33.096498 25736 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 16:11:33.101183 25751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:33.103183 25736 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260924 16:11:33.103425 25736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "3c68916d0f504e4b9bfebc15cdbffd4b"
format_stamp: "Formatted at 2026-09-24 16:11:33 on dist-test-slave-1gc3"
server_key: "fc585d37dae1c0ebf0f13c75866f06ba"
server_key_iv: "5d21d6eff642af33c6f73606735d0858"
server_key_version: "encryptionkey@0"
I20260924 16:11:33.103674 25736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:33.124963 25736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:33.129745 25736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:33.130079 25736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:33.151526 25736 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41191
I20260924 16:11:33.151618 25802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41191 every 8 connection(s)
I20260924 16:11:33.153066 25736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:33.158285 25803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:33.160969 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 25736
I20260924 16:11:33.161146 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:11:33.161648 3931 external_mini_cluster.cc:1428] Setting key d672771df0cbeac1dadb165fac452c90
I20260924 16:11:33.174003 25803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c68916d0f504e4b9bfebc15cdbffd4b: Bootstrap starting.
I20260924 16:11:33.178922 25803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c68916d0f504e4b9bfebc15cdbffd4b: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:33.179477 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:33.163955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:59803 (local address 127.0.0.1:41191)
0924 16:11:33.165253 (+ 1298us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:33.165274 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:33.165681 (+ 407us) server_negotiation.cc:407] Connection header received
0924 16:11:33.168005 (+ 2324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:33.168036 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:33.168574 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:33.168973 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:33.169822 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:33.171365 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:33.173633 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:33.174673 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:33.174867 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:33.176489 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:33.176499 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:33.176519 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:33.177194 (+ 675us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:33.177306 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:33.177336 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:33.177490 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:33.177872 (+ 382us) server_negotiation.cc:299] Negotiation successful
0924 16:11:33.178160 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":910,"thread_start_us":502,"threads_started":1}
I20260924 16:11:33.180682 25803 log.cc:824] T 00000000000000000000000000000000 P 3c68916d0f504e4b9bfebc15cdbffd4b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:33.185158 25803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c68916d0f504e4b9bfebc15cdbffd4b: No bootstrap required, opened a new log
I20260924 16:11:33.185212 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:11:33.195729 25803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c68916d0f504e4b9bfebc15cdbffd4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c68916d0f504e4b9bfebc15cdbffd4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41191 } }
I20260924 16:11:33.196429 25803 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c68916d0f504e4b9bfebc15cdbffd4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:33.196563 25803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c68916d0f504e4b9bfebc15cdbffd4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c68916d0f504e4b9bfebc15cdbffd4b, State: Initialized, Role: FOLLOWER
I20260924 16:11:33.197463 25803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c68916d0f504e4b9bfebc15cdbffd4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c68916d0f504e4b9bfebc15cdbffd4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41191 } }
I20260924 16:11:33.197826 25803 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c68916d0f504e4b9bfebc15cdbffd4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:33.198002 25803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c68916d0f504e4b9bfebc15cdbffd4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:33.198271 25803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c68916d0f504e4b9bfebc15cdbffd4b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:33.198699 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:33.190276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:60603 (local address 127.0.0.1:41191)
0924 16:11:33.190446 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:33.190451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:33.191119 (+ 668us) server_negotiation.cc:407] Connection header received
0924 16:11:33.191312 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:33.191319 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:33.191444 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:33.191577 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:33.192136 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:33.193415 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:33.195002 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:33.195683 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:33.195847 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:33.197691 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:33.197703 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:33.197708 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:33.197736 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:33.197820 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:33.197845 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:33.198029 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:33.198269 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 16:11:33.198332 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:11:33.201297 25803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c68916d0f504e4b9bfebc15cdbffd4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c68916d0f504e4b9bfebc15cdbffd4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41191 } }
I20260924 16:11:33.202047 25803 leader_election.cc:302] T 00000000000000000000000000000000 P 3c68916d0f504e4b9bfebc15cdbffd4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c68916d0f504e4b9bfebc15cdbffd4b; no voters:
I20260924 16:11:33.203014 25803 leader_election.cc:288] T 00000000000000000000000000000000 P 3c68916d0f504e4b9bfebc15cdbffd4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:33.203157 25813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c68916d0f504e4b9bfebc15cdbffd4b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:33.204010 25813 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c68916d0f504e4b9bfebc15cdbffd4b [term 1 LEADER]: Becoming Leader. State: Replica: 3c68916d0f504e4b9bfebc15cdbffd4b, State: Running, Role: LEADER
I20260924 16:11:33.204752 25813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c68916d0f504e4b9bfebc15cdbffd4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c68916d0f504e4b9bfebc15cdbffd4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41191 } }
I20260924 16:11:33.206163 25803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c68916d0f504e4b9bfebc15cdbffd4b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:33.209307 25814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c68916d0f504e4b9bfebc15cdbffd4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c68916d0f504e4b9bfebc15cdbffd4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c68916d0f504e4b9bfebc15cdbffd4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41191 } } }
I20260924 16:11:33.210180 25814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c68916d0f504e4b9bfebc15cdbffd4b [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:33.214512 25815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c68916d0f504e4b9bfebc15cdbffd4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c68916d0f504e4b9bfebc15cdbffd4b. Latest consensus state: current_term: 1 leader_uuid: "3c68916d0f504e4b9bfebc15cdbffd4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c68916d0f504e4b9bfebc15cdbffd4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41191 } } }
I20260924 16:11:33.215044 25815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c68916d0f504e4b9bfebc15cdbffd4b [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:33.220259 25822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:33.227465 25822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:33.239758 25822 catalog_manager.cc:1347] Generated new cluster ID: bbc28f74d3884a8781af3100e942b649
I20260924 16:11:33.239918 25822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:33.255859 25822 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:33.257561 25822 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:33.275556 25822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c68916d0f504e4b9bfebc15cdbffd4b: Generated new TSK 0
I20260924 16:11:33.276808 25822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:11:33.290926 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 25736
W20260924 16:11:33.292455 25798 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.27:59803
W20260924 16:11:33.293550 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 25736: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:11:33.293763 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 25736
W20260924 16:11:33.310426 25721 connection.cc:535] client connection to 127.0.0.1:41191 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41191): Connection reset by peer (error 104)
2026-09-24T16:11:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2054 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:33 dist-test-slave-1gc3 krb5kdc[25834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:33 dist-test-slave-1gc3 krb5kdc[25834](info): set up 2 sockets
Sep 24 16:11:33 dist-test-slave-1gc3 krb5kdc[25834](info): commencing operation
krb5kdc: starting...
W20260924 16:11:34.364329 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:11:34 dist-test-slave-1gc3 krb5kdc[25834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:11:34.695201 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36911
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:35.060197 25847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:35.060537 25847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:35.060658 25847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:35.068449 25847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:35.068620 25847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:35.068696 25847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:35.068814 25847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:35.068889 25847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:35.079861 25847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36911
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:35.081576 25847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:35.083043 25847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:35.096803 25855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:35.096992 25852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:35.096879 25853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:35.098163 25847 server_base.cc:1034] running on GCE node
I20260924 16:11:35.099040 25847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:35.100696 25847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:35.101984 25847 hybrid_clock.cc:648] HybridClock initialized: now 1790266295101901 us; error 71 us; skew 500 ppm
I20260924 16:11:35.102466 25847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:35 dist-test-slave-1gc3 krb5kdc[25834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:35.110239 25847 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:11:35.112185 25847 webserver.cc:460] Webserver started at http://127.0.0.1:46129/ using document root <none> and password file <none>
I20260924 16:11:35.112963 25847 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:35.113099 25847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:35.113420 25847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:35.116709 25847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "ef75a89bb5354c2a9afcd247c8014879"
format_stamp: "Formatted at 2026-09-24 16:11:35 on dist-test-slave-1gc3"
server_key: "78437ccee4934d3bda075d511710302a"
server_key_iv: "48c4779ba10be5daef0d70b3788f6831"
server_key_version: "encryptionkey@0"
I20260924 16:11:35.117535 25847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "ef75a89bb5354c2a9afcd247c8014879"
format_stamp: "Formatted at 2026-09-24 16:11:35 on dist-test-slave-1gc3"
server_key: "78437ccee4934d3bda075d511710302a"
server_key_iv: "48c4779ba10be5daef0d70b3788f6831"
server_key_version: "encryptionkey@0"
I20260924 16:11:35.123713 25847 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 16:11:35.127925 25862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:35.129911 25847 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:11:35.130194 25847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "ef75a89bb5354c2a9afcd247c8014879"
format_stamp: "Formatted at 2026-09-24 16:11:35 on dist-test-slave-1gc3"
server_key: "78437ccee4934d3bda075d511710302a"
server_key_iv: "48c4779ba10be5daef0d70b3788f6831"
server_key_version: "encryptionkey@0"
I20260924 16:11:35.130432 25847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:35.137313 25847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:35.142942 25847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:35.143306 25847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:35.164850 25847 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36911
I20260924 16:11:35.164939 25913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36911 every 8 connection(s)
I20260924 16:11:35.166778 25847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:35.169394 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 25847
I20260924 16:11:35.169598 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:11:35.170217 3931 external_mini_cluster.cc:1428] Setting key 526956e4ceb96711f02d777b3d3a1a00
I20260924 16:11:35.172029 25914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:35.180337 25917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:35.172765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:53185 (local address 127.0.0.1:36911)
0924 16:11:35.173745 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:35.173760 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:35.173815 (+ 55us) server_negotiation.cc:407] Connection header received
0924 16:11:35.175713 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:35.175746 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:35.176263 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:35.176745 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:35.178090 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:35.178136 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:35.178160 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:35.178193 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:35.178307 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:35.178340 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:35.178581 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:35.178735 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 16:11:35.179033 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":401,"threads_started":1}
I20260924 16:11:35.185235 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:11:35.187300 25914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef75a89bb5354c2a9afcd247c8014879: Bootstrap starting.
I20260924 16:11:35.192180 25914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef75a89bb5354c2a9afcd247c8014879: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:35.193776 25914 log.cc:824] T 00000000000000000000000000000000 P ef75a89bb5354c2a9afcd247c8014879: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:35.194896 25917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:35.191553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:34577 (local address 127.0.0.1:36911)
0924 16:11:35.191798 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:35.191804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:35.192116 (+ 312us) server_negotiation.cc:407] Connection header received
0924 16:11:35.192480 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:35.192486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:35.192567 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:35.192708 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:35.194009 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:35.194015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:35.194020 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:35.194045 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:35.194146 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:35.194176 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:35.194316 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:35.194439 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 16:11:35.194526 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 16:11:35.197471 25914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef75a89bb5354c2a9afcd247c8014879: No bootstrap required, opened a new log
I20260924 16:11:35.205927 25914 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef75a89bb5354c2a9afcd247c8014879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef75a89bb5354c2a9afcd247c8014879" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36911 } }
I20260924 16:11:35.206476 25914 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef75a89bb5354c2a9afcd247c8014879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:35.206593 25914 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef75a89bb5354c2a9afcd247c8014879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef75a89bb5354c2a9afcd247c8014879, State: Initialized, Role: FOLLOWER
I20260924 16:11:35.207347 25914 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef75a89bb5354c2a9afcd247c8014879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef75a89bb5354c2a9afcd247c8014879" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36911 } }
I20260924 16:11:35.207628 25914 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef75a89bb5354c2a9afcd247c8014879 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:35.207778 25914 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef75a89bb5354c2a9afcd247c8014879 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:35.207969 25914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef75a89bb5354c2a9afcd247c8014879 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:35.210353 25914 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef75a89bb5354c2a9afcd247c8014879 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef75a89bb5354c2a9afcd247c8014879" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36911 } }
I20260924 16:11:35.210952 25914 leader_election.cc:302] T 00000000000000000000000000000000 P ef75a89bb5354c2a9afcd247c8014879 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef75a89bb5354c2a9afcd247c8014879; no voters:
I20260924 16:11:35.211685 25914 leader_election.cc:288] T 00000000000000000000000000000000 P ef75a89bb5354c2a9afcd247c8014879 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:35.212077 25924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef75a89bb5354c2a9afcd247c8014879 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:35.212961 25924 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef75a89bb5354c2a9afcd247c8014879 [term 1 LEADER]: Becoming Leader. State: Replica: ef75a89bb5354c2a9afcd247c8014879, State: Running, Role: LEADER
I20260924 16:11:35.213930 25924 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef75a89bb5354c2a9afcd247c8014879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef75a89bb5354c2a9afcd247c8014879" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36911 } }
I20260924 16:11:35.214684 25914 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef75a89bb5354c2a9afcd247c8014879 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:35.218504 25926 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef75a89bb5354c2a9afcd247c8014879 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef75a89bb5354c2a9afcd247c8014879. Latest consensus state: current_term: 1 leader_uuid: "ef75a89bb5354c2a9afcd247c8014879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef75a89bb5354c2a9afcd247c8014879" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36911 } } }
I20260924 16:11:35.218504 25925 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef75a89bb5354c2a9afcd247c8014879 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef75a89bb5354c2a9afcd247c8014879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef75a89bb5354c2a9afcd247c8014879" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36911 } } }
I20260924 16:11:35.219305 25926 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef75a89bb5354c2a9afcd247c8014879 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:35.219318 25925 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef75a89bb5354c2a9afcd247c8014879 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:35.224160 25933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:35.230654 25933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:35.242338 25933 catalog_manager.cc:1347] Generated new cluster ID: c2fb2dd007274ea9835a38eb7a1c2140
I20260924 16:11:35.242532 25933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:35.273653 25933 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:35.274544 25933 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:35.291432 25933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef75a89bb5354c2a9afcd247c8014879: Generated new TSK 0
I20260924 16:11:35.292639 25933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:11:35.301146 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 25847
W20260924 16:11:35.303333 25909 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.27:53185
W20260924 16:11:35.304332 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 25847: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:11:35.304564 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 25847
2026-09-24T16:11:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2011 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:35 dist-test-slave-1gc3 krb5kdc[25945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:35 dist-test-slave-1gc3 krb5kdc[25945](info): set up 2 sockets
Sep 24 16:11:35 dist-test-slave-1gc3 krb5kdc[25945](info): commencing operation
krb5kdc: starting...
W20260924 16:11:36.384428 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.002s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:11:36 dist-test-slave-1gc3 krb5kdc[25945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:11:36.710182 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:37781
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:37781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:37.128504 25958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:37.128927 25958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:37.129081 25958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:37.137552 25958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:37.137722 25958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:37.137802 25958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:37.137894 25958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:37.137957 25958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:37.149709 25958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:37781
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:37781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:37.151764 25958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:37.153571 25958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:37.167961 25963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:37.167969 25964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:37.167994 25966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:37.169842 25958 server_base.cc:1034] running on GCE node
I20260924 16:11:37.171032 25958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:37.173159 25958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:37.174448 25958 hybrid_clock.cc:648] HybridClock initialized: now 1790266297174381 us; error 82 us; skew 500 ppm
I20260924 16:11:37.174926 25958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:37 dist-test-slave-1gc3 krb5kdc[25945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266297, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:37.178776 25958 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:11:37.180707 25958 webserver.cc:460] Webserver started at http://127.3.214.254:34127/ using document root <none> and password file <none>
I20260924 16:11:37.181592 25958 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:37.181730 25958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:37.182132 25958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:37.185896 25958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "5b8f4a336fa84adfa3988035cd57c201"
format_stamp: "Formatted at 2026-09-24 16:11:37 on dist-test-slave-1gc3"
server_key: "dc93be9fe45e4b5ebe85eb1ce32122ed"
server_key_iv: "b65e3eccce9f6209e7f5f123f2eb5e68"
server_key_version: "encryptionkey@0"
I20260924 16:11:37.186988 25958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "5b8f4a336fa84adfa3988035cd57c201"
format_stamp: "Formatted at 2026-09-24 16:11:37 on dist-test-slave-1gc3"
server_key: "dc93be9fe45e4b5ebe85eb1ce32122ed"
server_key_iv: "b65e3eccce9f6209e7f5f123f2eb5e68"
server_key_version: "encryptionkey@0"
I20260924 16:11:37.194072 25958 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s
I20260924 16:11:37.199090 25973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:37.200965 25958 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260924 16:11:37.201226 25958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "5b8f4a336fa84adfa3988035cd57c201"
format_stamp: "Formatted at 2026-09-24 16:11:37 on dist-test-slave-1gc3"
server_key: "dc93be9fe45e4b5ebe85eb1ce32122ed"
server_key_iv: "b65e3eccce9f6209e7f5f123f2eb5e68"
server_key_version: "encryptionkey@0"
I20260924 16:11:37.201493 25958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:37.234195 25958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:37.239624 25958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:37.239970 25958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:37.262859 25958 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:37781
I20260924 16:11:37.262868 26024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:37781 every 8 connection(s)
I20260924 16:11:37.264618 25958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:37.266129 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 25958
I20260924 16:11:37.266355 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:11:37.266917 3931 external_mini_cluster.cc:1428] Setting key f6b994b5ce74617494afc136c90b08c7
I20260924 16:11:37.270016 26025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 16:11:37 dist-test-slave-1gc3 krb5kdc[25945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:11:37.284464 26025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b8f4a336fa84adfa3988035cd57c201: Bootstrap starting.
I20260924 16:11:37.288209 26025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b8f4a336fa84adfa3988035cd57c201: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:37.289441 26025 log.cc:824] T 00000000000000000000000000000000 P 5b8f4a336fa84adfa3988035cd57c201: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:37.292714 26025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b8f4a336fa84adfa3988035cd57c201: No bootstrap required, opened a new log
I20260924 16:11:37.295214 26028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:37.269310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56597 (local address 127.3.214.254:37781)
0924 16:11:37.270427 (+ 1117us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:37.270443 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:37.270499 (+ 56us) server_negotiation.cc:407] Connection header received
0924 16:11:37.272529 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:37.272561 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:37.273158 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:37.273639 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:37.275715 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:37.277430 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:37.279329 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:37.280247 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:37.280429 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:37.286005 (+ 5576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:37.286016 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:37.286045 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:37.286156 (+ 111us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:37.289782 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:37.290910 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:37.290920 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:37.290931 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:37.291167 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:37.291677 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:37.291681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:37.291683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:37.292365 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:37.292759 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:37.293308 (+ 549us) server_negotiation.cc:299] Negotiation successful
0924 16:11:37.293750 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":470,"threads_started":1}
I20260924 16:11:37.301059 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:11:37.302547 26025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b8f4a336fa84adfa3988035cd57c201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8f4a336fa84adfa3988035cd57c201" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 37781 } }
I20260924 16:11:37.302994 26025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b8f4a336fa84adfa3988035cd57c201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:37.303089 26025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b8f4a336fa84adfa3988035cd57c201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8f4a336fa84adfa3988035cd57c201, State: Initialized, Role: FOLLOWER
I20260924 16:11:37.303721 26025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b8f4a336fa84adfa3988035cd57c201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8f4a336fa84adfa3988035cd57c201" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 37781 } }
I20260924 16:11:37.304056 26025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b8f4a336fa84adfa3988035cd57c201 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:37.304234 26025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b8f4a336fa84adfa3988035cd57c201 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:37.304425 26025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b8f4a336fa84adfa3988035cd57c201 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:37.307477 26025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b8f4a336fa84adfa3988035cd57c201 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8f4a336fa84adfa3988035cd57c201" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 37781 } }
I20260924 16:11:37.308126 26025 leader_election.cc:302] T 00000000000000000000000000000000 P 5b8f4a336fa84adfa3988035cd57c201 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b8f4a336fa84adfa3988035cd57c201; no voters:
I20260924 16:11:37.308815 26025 leader_election.cc:288] T 00000000000000000000000000000000 P 5b8f4a336fa84adfa3988035cd57c201 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:37.308925 26035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b8f4a336fa84adfa3988035cd57c201 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:37.310443 26035 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b8f4a336fa84adfa3988035cd57c201 [term 1 LEADER]: Becoming Leader. State: Replica: 5b8f4a336fa84adfa3988035cd57c201, State: Running, Role: LEADER
I20260924 16:11:37.311224 26035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b8f4a336fa84adfa3988035cd57c201 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8f4a336fa84adfa3988035cd57c201" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 37781 } }
I20260924 16:11:37.311648 26025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b8f4a336fa84adfa3988035cd57c201 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:37.319238 26036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b8f4a336fa84adfa3988035cd57c201 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b8f4a336fa84adfa3988035cd57c201" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8f4a336fa84adfa3988035cd57c201" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 37781 } } }
I20260924 16:11:37.319885 26036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b8f4a336fa84adfa3988035cd57c201 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:37.319324 26037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b8f4a336fa84adfa3988035cd57c201 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b8f4a336fa84adfa3988035cd57c201. Latest consensus state: current_term: 1 leader_uuid: "5b8f4a336fa84adfa3988035cd57c201" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8f4a336fa84adfa3988035cd57c201" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 37781 } } }
I20260924 16:11:37.320216 26037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b8f4a336fa84adfa3988035cd57c201 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:37.322837 26042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:37.325363 26028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:37.306458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44047 (local address 127.3.214.254:37781)
0924 16:11:37.306689 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:37.306697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:37.307686 (+ 989us) server_negotiation.cc:407] Connection header received
0924 16:11:37.307901 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:37.307909 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:37.307997 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:37.308097 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:37.310068 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:37.311285 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:37.312722 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:37.313445 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:37.313600 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:37.317418 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:37.317425 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:37.317428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:37.317470 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:37.321322 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:37.322587 (+ 1265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:37.322593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:37.322595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:37.322790 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:37.323399 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:37.323403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:37.323406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:37.323777 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:37.324031 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:37.324635 (+ 604us) server_negotiation.cc:299] Negotiation successful
0924 16:11:37.324867 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:11:37.329555 26042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:37.344432 26042 catalog_manager.cc:1347] Generated new cluster ID: 1764a213123e4611bc1a75f0b137061b
I20260924 16:11:37.344792 26042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:37.360178 26042 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:37.362268 26042 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:37.374300 26042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b8f4a336fa84adfa3988035cd57c201: Generated new TSK 0
I20260924 16:11:37.375610 26042 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:11:37.389796 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 25958
I20260924 16:11:37.392295 26020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:37.498543 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 25958
W20260924 16:11:37.515204 25943 connection.cc:535] client connection to 127.3.214.254:37781 recv error: Network error: failed to read from TLS socket (remote: 127.3.214.254:37781): Connection reset by peer (error 104)
2026-09-24T16:11:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2193 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:37 dist-test-slave-1gc3 krb5kdc[26057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:37 dist-test-slave-1gc3 krb5kdc[26057](info): set up 2 sockets
Sep 24 16:11:37 dist-test-slave-1gc3 krb5kdc[26057](info): commencing operation
krb5kdc: starting...
W20260924 16:11:38.556030 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:11:38 dist-test-slave-1gc3 krb5kdc[26057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:11:38.887394 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:45395
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:45395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:39.299245 26070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:39.299661 26070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:39.299767 26070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:39.308326 26070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:39.308494 26070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:39.308547 26070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:39.308629 26070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:39.308681 26070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:39.320127 26070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:45395
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:45395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:39.321871 26070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:39.323485 26070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:39.338727 26075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:39.338639 26076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:39.339213 26078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:39.340143 26070 server_base.cc:1034] running on GCE node
I20260924 16:11:39.341269 26070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:39.343295 26070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:39.344622 26070 hybrid_clock.cc:648] HybridClock initialized: now 1790266299344533 us; error 76 us; skew 500 ppm
I20260924 16:11:39.345104 26070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:39 dist-test-slave-1gc3 krb5kdc[26057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266299, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:39.348917 26070 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:11:39.350934 26070 webserver.cc:460] Webserver started at http://127.3.214.254:42493/ using document root <none> and password file <none>
I20260924 16:11:39.351756 26070 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:39.351888 26070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:39.352224 26070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:39.355676 26070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "4c67885265674d70935eed20bfbcca33"
format_stamp: "Formatted at 2026-09-24 16:11:39 on dist-test-slave-1gc3"
server_key: "6ffda51df1e07260cafa3edbac447d69"
server_key_iv: "f72f11ea7cd1e8260a927345fd5115a1"
server_key_version: "encryptionkey@0"
I20260924 16:11:39.356571 26070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "4c67885265674d70935eed20bfbcca33"
format_stamp: "Formatted at 2026-09-24 16:11:39 on dist-test-slave-1gc3"
server_key: "6ffda51df1e07260cafa3edbac447d69"
server_key_iv: "f72f11ea7cd1e8260a927345fd5115a1"
server_key_version: "encryptionkey@0"
I20260924 16:11:39.363742 26070 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260924 16:11:39.368333 26085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:39.370194 26070 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:11:39.370453 26070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "4c67885265674d70935eed20bfbcca33"
format_stamp: "Formatted at 2026-09-24 16:11:39 on dist-test-slave-1gc3"
server_key: "6ffda51df1e07260cafa3edbac447d69"
server_key_iv: "f72f11ea7cd1e8260a927345fd5115a1"
server_key_version: "encryptionkey@0"
I20260924 16:11:39.370682 26070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:39.387820 26070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:39.392449 26070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:39.392750 26070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:39.415088 26070 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:45395
I20260924 16:11:39.415143 26136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:45395 every 8 connection(s)
I20260924 16:11:39.416657 26070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:39.421741 26137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:39.422616 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 26070
I20260924 16:11:39.422837 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:11:39.423527 3931 external_mini_cluster.cc:1428] Setting key 45d78f37dbca584ae0d014f1866e5743
I20260924 16:11:39.438699 26137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c67885265674d70935eed20bfbcca33: Bootstrap starting.
I20260924 16:11:39.439723 26140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:39.426080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59103 (local address 127.3.214.254:45395)
0924 16:11:39.427242 (+ 1162us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:39.427258 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:39.427308 (+ 50us) server_negotiation.cc:407] Connection header received
0924 16:11:39.429707 (+ 2399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:39.429747 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:39.430359 (+ 612us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:39.430810 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:39.431456 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:39.432828 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:39.434313 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:39.435233 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:39.435444 (+ 211us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:39.437023 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:39.437032 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:39.437053 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:39.437683 (+ 630us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:39.437783 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:39.437813 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:39.437979 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:39.438376 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 16:11:39.438566 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":502,"threads_started":1}
I20260924 16:11:39.443363 26137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c67885265674d70935eed20bfbcca33: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:39.444981 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:11:39.445109 26137 log.cc:824] T 00000000000000000000000000000000 P 4c67885265674d70935eed20bfbcca33: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:39.449183 26137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c67885265674d70935eed20bfbcca33: No bootstrap required, opened a new log
I20260924 16:11:39.458050 26140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:39.450167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52873 (local address 127.3.214.254:45395)
0924 16:11:39.450390 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:39.450397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:39.450731 (+ 334us) server_negotiation.cc:407] Connection header received
0924 16:11:39.450977 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:39.450982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:39.451059 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:39.451171 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:39.451931 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:39.452798 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:39.454900 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:39.455466 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:39.455576 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:39.457080 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:39.457085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:39.457089 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:39.457110 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:39.457155 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:39.457176 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:39.457274 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:39.457491 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 16:11:39.457545 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 16:11:39.460906 26137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c67885265674d70935eed20bfbcca33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c67885265674d70935eed20bfbcca33" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 45395 } }
I20260924 16:11:39.461433 26137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c67885265674d70935eed20bfbcca33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:39.461541 26137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c67885265674d70935eed20bfbcca33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c67885265674d70935eed20bfbcca33, State: Initialized, Role: FOLLOWER
I20260924 16:11:39.462332 26137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c67885265674d70935eed20bfbcca33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c67885265674d70935eed20bfbcca33" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 45395 } }
I20260924 16:11:39.462615 26137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c67885265674d70935eed20bfbcca33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:39.462779 26137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c67885265674d70935eed20bfbcca33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:39.462991 26137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c67885265674d70935eed20bfbcca33 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:39.465919 26137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c67885265674d70935eed20bfbcca33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c67885265674d70935eed20bfbcca33" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 45395 } }
I20260924 16:11:39.466696 26137 leader_election.cc:302] T 00000000000000000000000000000000 P 4c67885265674d70935eed20bfbcca33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c67885265674d70935eed20bfbcca33; no voters:
I20260924 16:11:39.467504 26137 leader_election.cc:288] T 00000000000000000000000000000000 P 4c67885265674d70935eed20bfbcca33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:39.467640 26147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c67885265674d70935eed20bfbcca33 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:39.468451 26147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c67885265674d70935eed20bfbcca33 [term 1 LEADER]: Becoming Leader. State: Replica: 4c67885265674d70935eed20bfbcca33, State: Running, Role: LEADER
I20260924 16:11:39.469301 26147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c67885265674d70935eed20bfbcca33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c67885265674d70935eed20bfbcca33" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 45395 } }
I20260924 16:11:39.470479 26137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c67885265674d70935eed20bfbcca33 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:39.473757 26148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c67885265674d70935eed20bfbcca33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c67885265674d70935eed20bfbcca33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c67885265674d70935eed20bfbcca33" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 45395 } } }
I20260924 16:11:39.474279 26148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c67885265674d70935eed20bfbcca33 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:39.474293 26149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c67885265674d70935eed20bfbcca33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c67885265674d70935eed20bfbcca33. Latest consensus state: current_term: 1 leader_uuid: "4c67885265674d70935eed20bfbcca33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c67885265674d70935eed20bfbcca33" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 45395 } } }
I20260924 16:11:39.480726 26149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c67885265674d70935eed20bfbcca33 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:39.482007 26155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:39.487988 26155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:39.499529 26155 catalog_manager.cc:1347] Generated new cluster ID: 41a76f68213c493fb88792e4e4d7c36a
I20260924 16:11:39.499729 26155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:39.516920 26155 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:39.518833 26155 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:39.529387 26155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c67885265674d70935eed20bfbcca33: Generated new TSK 0
I20260924 16:11:39.530520 26155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:11:39.543294 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 26070
W20260924 16:11:39.544857 26132 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59103
W20260924 16:11:39.545861 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 26070: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:11:39.546013 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 26070
W20260924 16:11:39.562296 26055 connection.cc:535] client connection to 127.3.214.254:45395 recv error: Network error: failed to read from TLS socket (remote: 127.3.214.254:45395): Connection reset by peer (error 104)
2026-09-24T16:11:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2046 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:39 dist-test-slave-1gc3 krb5kdc[26168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:39 dist-test-slave-1gc3 krb5kdc[26168](info): set up 2 sockets
Sep 24 16:11:39 dist-test-slave-1gc3 krb5kdc[26168](info): commencing operation
krb5kdc: starting...
W20260924 16:11:40.608698 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:11:40 dist-test-slave-1gc3 krb5kdc[26168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:11:40.946211 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:42191
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:42191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:41.371520 26181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:41.371940 26181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:41.372047 26181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:41.380635 26181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:41.380797 26181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:41.380851 26181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:41.380920 26181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:41.380970 26181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:41.392669 26181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:42191
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:42191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:41.394510 26181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:41.396229 26181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:41.411114 26189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:41.411198 26187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:41.411202 26186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:41.417651 26181 server_base.cc:1034] running on GCE node
I20260924 16:11:41.418907 26181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:41.420948 26181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:41.422268 26181 hybrid_clock.cc:648] HybridClock initialized: now 1790266301422190 us; error 61 us; skew 500 ppm
I20260924 16:11:41.422739 26181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:41 dist-test-slave-1gc3 krb5kdc[26168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266301, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:41.429632 26181 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:11:41.431809 26181 webserver.cc:460] Webserver started at http://127.3.214.254:34553/ using document root <none> and password file <none>
I20260924 16:11:41.432754 26181 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:41.432873 26181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:41.433211 26181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:41.437261 26181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "7400de10be1346fd9ed5b024b2336e5f"
format_stamp: "Formatted at 2026-09-24 16:11:41 on dist-test-slave-1gc3"
server_key: "df26ff0139307b83c19af4733622b450"
server_key_iv: "b1a5e19ae65747869d2b9c05d23f23be"
server_key_version: "encryptionkey@0"
I20260924 16:11:41.438345 26181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "7400de10be1346fd9ed5b024b2336e5f"
format_stamp: "Formatted at 2026-09-24 16:11:41 on dist-test-slave-1gc3"
server_key: "df26ff0139307b83c19af4733622b450"
server_key_iv: "b1a5e19ae65747869d2b9c05d23f23be"
server_key_version: "encryptionkey@0"
I20260924 16:11:41.445583 26181 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260924 16:11:41.450839 26196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:41.452929 26181 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260924 16:11:41.453202 26181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "7400de10be1346fd9ed5b024b2336e5f"
format_stamp: "Formatted at 2026-09-24 16:11:41 on dist-test-slave-1gc3"
server_key: "df26ff0139307b83c19af4733622b450"
server_key_iv: "b1a5e19ae65747869d2b9c05d23f23be"
server_key_version: "encryptionkey@0"
I20260924 16:11:41.453459 26181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:41.461373 26181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:41.467450 26181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:41.467880 26181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:41.492349 26181 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:42191
I20260924 16:11:41.492374 26247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:42191 every 8 connection(s)
I20260924 16:11:41.494451 26181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:41.499346 26248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:41.502507 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 26181
I20260924 16:11:41.502722 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:11:41.503317 3931 external_mini_cluster.cc:1428] Setting key f50cd52b131a51a9ebb0de591c089e7a
I20260924 16:11:41.514147 26248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7400de10be1346fd9ed5b024b2336e5f: Bootstrap starting.
I20260924 16:11:41.514545 26251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:41.505940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33681 (local address 127.3.214.254:42191)
0924 16:11:41.507128 (+ 1188us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:41.507151 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:41.507218 (+ 67us) server_negotiation.cc:407] Connection header received
0924 16:11:41.509471 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:41.509513 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:41.510202 (+ 689us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:41.510835 (+ 633us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:41.512278 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:41.512287 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:41.512309 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:41.512327 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:41.512398 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:41.512423 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:41.512710 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:41.512879 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 16:11:41.513151 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":862,"thread_start_us":480,"threads_started":1}
I20260924 16:11:41.518286 26248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7400de10be1346fd9ed5b024b2336e5f: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:41.519677 26248 log.cc:824] T 00000000000000000000000000000000 P 7400de10be1346fd9ed5b024b2336e5f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:41.519691 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:11:41.523674 26248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7400de10be1346fd9ed5b024b2336e5f: No bootstrap required, opened a new log
I20260924 16:11:41.528600 26251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:41.525414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41445 (local address 127.3.214.254:42191)
0924 16:11:41.525691 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:41.525699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:41.526139 (+ 440us) server_negotiation.cc:407] Connection header received
0924 16:11:41.526358 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:41.526368 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:41.526445 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:41.526607 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:41.527745 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:41.527750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:41.527754 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:41.527779 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:41.527832 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:41.527852 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:41.527975 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:41.528131 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 16:11:41.528204 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260924 16:11:41.533281 26248 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7400de10be1346fd9ed5b024b2336e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7400de10be1346fd9ed5b024b2336e5f" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42191 } }
I20260924 16:11:41.533720 26248 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7400de10be1346fd9ed5b024b2336e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:41.533831 26248 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7400de10be1346fd9ed5b024b2336e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7400de10be1346fd9ed5b024b2336e5f, State: Initialized, Role: FOLLOWER
I20260924 16:11:41.534520 26248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7400de10be1346fd9ed5b024b2336e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7400de10be1346fd9ed5b024b2336e5f" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42191 } }
I20260924 16:11:41.534758 26248 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7400de10be1346fd9ed5b024b2336e5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:41.534915 26248 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7400de10be1346fd9ed5b024b2336e5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:41.535079 26248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7400de10be1346fd9ed5b024b2336e5f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:41.537403 26248 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7400de10be1346fd9ed5b024b2336e5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7400de10be1346fd9ed5b024b2336e5f" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42191 } }
I20260924 16:11:41.538007 26248 leader_election.cc:302] T 00000000000000000000000000000000 P 7400de10be1346fd9ed5b024b2336e5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7400de10be1346fd9ed5b024b2336e5f; no voters:
I20260924 16:11:41.538782 26248 leader_election.cc:288] T 00000000000000000000000000000000 P 7400de10be1346fd9ed5b024b2336e5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:41.539057 26258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7400de10be1346fd9ed5b024b2336e5f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:41.540501 26258 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7400de10be1346fd9ed5b024b2336e5f [term 1 LEADER]: Becoming Leader. State: Replica: 7400de10be1346fd9ed5b024b2336e5f, State: Running, Role: LEADER
I20260924 16:11:41.541412 26258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7400de10be1346fd9ed5b024b2336e5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7400de10be1346fd9ed5b024b2336e5f" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42191 } }
I20260924 16:11:41.541620 26248 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7400de10be1346fd9ed5b024b2336e5f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:41.546290 26259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7400de10be1346fd9ed5b024b2336e5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7400de10be1346fd9ed5b024b2336e5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7400de10be1346fd9ed5b024b2336e5f" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42191 } } }
I20260924 16:11:41.546268 26260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7400de10be1346fd9ed5b024b2336e5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7400de10be1346fd9ed5b024b2336e5f. Latest consensus state: current_term: 1 leader_uuid: "7400de10be1346fd9ed5b024b2336e5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7400de10be1346fd9ed5b024b2336e5f" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42191 } } }
I20260924 16:11:41.547013 26260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7400de10be1346fd9ed5b024b2336e5f [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:41.547013 26259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7400de10be1346fd9ed5b024b2336e5f [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:41.549620 26267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:41.555511 26267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:41.568454 26267 catalog_manager.cc:1347] Generated new cluster ID: 23d3cce28f5e42af863991380980b32a
I20260924 16:11:41.568598 26267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:41.588295 26267 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:41.588944 26267 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:41.606593 26267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7400de10be1346fd9ed5b024b2336e5f: Generated new TSK 0
I20260924 16:11:41.607829 26267 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:11:41.618633 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 26181
W20260924 16:11:41.620414 26243 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33681
W20260924 16:11:41.621609 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 26181: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:11:41.621884 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 26181
2026-09-24T16:11:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2076 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:41 dist-test-slave-1gc3 krb5kdc[26279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:41 dist-test-slave-1gc3 krb5kdc[26279](info): set up 2 sockets
Sep 24 16:11:41 dist-test-slave-1gc3 krb5kdc[26279](info): commencing operation
krb5kdc: starting...
W20260924 16:11:42.696250 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.003s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:11:42 dist-test-slave-1gc3 krb5kdc[26279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:11:43.018801 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:42363
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:42363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:43.438495 26292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:43.438964 26292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:43.439113 26292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:43.447670 26292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:43.447873 26292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:43.447959 26292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:43.448051 26292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:43.448113 26292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:43.460033 26292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:42363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:42363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:43.461931 26292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:43.463753 26292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:43.478564 26298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:43.478658 26300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:43.478652 26297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:43.479875 26292 server_base.cc:1034] running on GCE node
I20260924 16:11:43.481091 26292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:43.483122 26292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:43.484411 26292 hybrid_clock.cc:648] HybridClock initialized: now 1790266303484353 us; error 52 us; skew 500 ppm
I20260924 16:11:43.484867 26292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:43 dist-test-slave-1gc3 krb5kdc[26279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266303, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:43.488258 26292 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:11:43.490085 26292 webserver.cc:460] Webserver started at http://127.3.214.254:44757/ using document root <none> and password file <none>
I20260924 16:11:43.491189 26292 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:43.491377 26292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:43.491822 26292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:43.495496 26292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "553b97f8766845f48897048dadebed60"
format_stamp: "Formatted at 2026-09-24 16:11:43 on dist-test-slave-1gc3"
server_key: "8b7f7e4b629d56cdb0ec7325fac1bf04"
server_key_iv: "15dae1fed6e0158e60a5c8b53eb20471"
server_key_version: "encryptionkey@0"
I20260924 16:11:43.496316 26292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "553b97f8766845f48897048dadebed60"
format_stamp: "Formatted at 2026-09-24 16:11:43 on dist-test-slave-1gc3"
server_key: "8b7f7e4b629d56cdb0ec7325fac1bf04"
server_key_iv: "15dae1fed6e0158e60a5c8b53eb20471"
server_key_version: "encryptionkey@0"
I20260924 16:11:43.502679 26292 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20260924 16:11:43.507387 26307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:43.509393 26292 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:11:43.509653 26292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "553b97f8766845f48897048dadebed60"
format_stamp: "Formatted at 2026-09-24 16:11:43 on dist-test-slave-1gc3"
server_key: "8b7f7e4b629d56cdb0ec7325fac1bf04"
server_key_iv: "15dae1fed6e0158e60a5c8b53eb20471"
server_key_version: "encryptionkey@0"
I20260924 16:11:43.509903 26292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:43.529927 26292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:43.535368 26292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:43.535691 26292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:43.559101 26292 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:42363
I20260924 16:11:43.559146 26358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:42363 every 8 connection(s)
I20260924 16:11:43.560664 26292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:43.564388 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 26292
I20260924 16:11:43.564634 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:11:43.565308 3931 external_mini_cluster.cc:1428] Setting key a155546148b77ce79ac6590fd0eb952e
I20260924 16:11:43.566898 26359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 16:11:43 dist-test-slave-1gc3 krb5kdc[26279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:11:43.583885 26359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 553b97f8766845f48897048dadebed60: Bootstrap starting.
I20260924 16:11:43.589162 26359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 553b97f8766845f48897048dadebed60: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:43.590888 26359 log.cc:824] T 00000000000000000000000000000000 P 553b97f8766845f48897048dadebed60: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:43.592854 26362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:43.567920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35491 (local address 127.3.214.254:42363)
0924 16:11:43.569066 (+ 1146us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:43.569085 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:43.569141 (+ 56us) server_negotiation.cc:407] Connection header received
0924 16:11:43.571343 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:43.571383 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:43.572014 (+ 631us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:43.572490 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:43.574221 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:43.575739 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:43.577125 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:43.578283 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:43.578515 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:43.583470 (+ 4955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:43.583480 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:43.583501 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:43.583560 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:43.586992 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:43.588373 (+ 1381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:43.588391 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:43.588401 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:43.588621 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:43.589100 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:43.589103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:43.589105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:43.589830 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:43.590092 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:43.590775 (+ 683us) server_negotiation.cc:299] Negotiation successful
0924 16:11:43.591265 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":475,"threads_started":1}
I20260924 16:11:43.595297 26359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 553b97f8766845f48897048dadebed60: No bootstrap required, opened a new log
I20260924 16:11:43.597601 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:11:43.605108 26359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 553b97f8766845f48897048dadebed60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "553b97f8766845f48897048dadebed60" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42363 } }
I20260924 16:11:43.605657 26359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 553b97f8766845f48897048dadebed60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:43.605782 26359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 553b97f8766845f48897048dadebed60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 553b97f8766845f48897048dadebed60, State: Initialized, Role: FOLLOWER
I20260924 16:11:43.606679 26359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 553b97f8766845f48897048dadebed60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "553b97f8766845f48897048dadebed60" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42363 } }
I20260924 16:11:43.607165 26359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 553b97f8766845f48897048dadebed60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:43.607342 26359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 553b97f8766845f48897048dadebed60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:43.607553 26359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 553b97f8766845f48897048dadebed60 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:43.610616 26359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 553b97f8766845f48897048dadebed60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "553b97f8766845f48897048dadebed60" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42363 } }
I20260924 16:11:43.611406 26359 leader_election.cc:302] T 00000000000000000000000000000000 P 553b97f8766845f48897048dadebed60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 553b97f8766845f48897048dadebed60; no voters:
I20260924 16:11:43.612301 26359 leader_election.cc:288] T 00000000000000000000000000000000 P 553b97f8766845f48897048dadebed60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:43.612495 26369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 553b97f8766845f48897048dadebed60 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:43.615756 26359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 553b97f8766845f48897048dadebed60 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:43.616648 26369 raft_consensus.cc:686] T 00000000000000000000000000000000 P 553b97f8766845f48897048dadebed60 [term 1 LEADER]: Becoming Leader. State: Replica: 553b97f8766845f48897048dadebed60, State: Running, Role: LEADER
I20260924 16:11:43.617624 26369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 553b97f8766845f48897048dadebed60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "553b97f8766845f48897048dadebed60" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42363 } }
I20260924 16:11:43.621049 26371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 553b97f8766845f48897048dadebed60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 553b97f8766845f48897048dadebed60. Latest consensus state: current_term: 1 leader_uuid: "553b97f8766845f48897048dadebed60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "553b97f8766845f48897048dadebed60" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42363 } } }
I20260924 16:11:43.621521 26371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 553b97f8766845f48897048dadebed60 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:43.622244 26362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:43.602980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57523 (local address 127.3.214.254:42363)
0924 16:11:43.603192 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:43.603199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:43.603724 (+ 525us) server_negotiation.cc:407] Connection header received
0924 16:11:43.603914 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:43.603918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:43.603995 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:43.604083 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:43.605678 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:43.606651 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:43.608201 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:43.608766 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:43.608895 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:43.611778 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:43.611808 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:43.611815 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:43.611877 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:43.616252 (+ 4375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:43.617480 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:43.617488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:43.617491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:43.617764 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:43.618335 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:43.618341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:43.618344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:43.618822 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:43.619171 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:43.621324 (+ 2153us) server_negotiation.cc:299] Negotiation successful
0924 16:11:43.621671 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 16:11:43.622182 26370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 553b97f8766845f48897048dadebed60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "553b97f8766845f48897048dadebed60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "553b97f8766845f48897048dadebed60" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42363 } } }
I20260924 16:11:43.622521 26370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 553b97f8766845f48897048dadebed60 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:43.633741 26379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:43.639384 26379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:43.651759 26379 catalog_manager.cc:1347] Generated new cluster ID: fd133adef55d44e5b7ae5675d832df39
I20260924 16:11:43.651997 26379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:43.674460 26379 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:43.676656 26379 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:43.695755 26379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 553b97f8766845f48897048dadebed60: Generated new TSK 0
I20260924 16:11:43.697387 26379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:11:43.718497 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 26292
I20260924 16:11:43.720320 26354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:43.828272 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 26292
2026-09-24T16:11:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2207 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:43 dist-test-slave-1gc3 krb5kdc[26392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:43 dist-test-slave-1gc3 krb5kdc[26392](info): set up 2 sockets
Sep 24 16:11:43 dist-test-slave-1gc3 krb5kdc[26392](info): commencing operation
krb5kdc: starting...
W20260924 16:11:44.896571 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:11:45 dist-test-slave-1gc3 krb5kdc[26392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:11:45.226572 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:40689
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:46859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:40689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:45.623530 26405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:45.623862 26405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:45.623960 26405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:45.631621 26405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:45.631767 26405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:45.631817 26405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:45.631891 26405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:45.631942 26405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:45.643126 26405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:46859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:40689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:40689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:45.644577 26405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:45.645998 26405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:45.659993 26410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:45.659945 26413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:45.660014 26411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:45.661147 26405 server_base.cc:1034] running on GCE node
I20260924 16:11:45.662077 26405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:45.663880 26405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:45.665205 26405 hybrid_clock.cc:648] HybridClock initialized: now 1790266305665123 us; error 73 us; skew 500 ppm
I20260924 16:11:45.665657 26405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:45 dist-test-slave-1gc3 krb5kdc[26392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266305, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:45.669392 26405 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:11:45.671320 26405 webserver.cc:460] Webserver started at http://127.3.214.254:38007/ using document root <none> and password file <none>
I20260924 16:11:45.672098 26405 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:45.672242 26405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:45.672554 26405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:45.675818 26405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "971fcac06fc145858d467b5026ffe6a8"
format_stamp: "Formatted at 2026-09-24 16:11:45 on dist-test-slave-1gc3"
server_key: "9be8397596a29520cefc0fcd2595938e"
server_key_iv: "bf6db572cd52fac889712ed738680fc0"
server_key_version: "encryptionkey@0"
I20260924 16:11:45.676646 26405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "971fcac06fc145858d467b5026ffe6a8"
format_stamp: "Formatted at 2026-09-24 16:11:45 on dist-test-slave-1gc3"
server_key: "9be8397596a29520cefc0fcd2595938e"
server_key_iv: "bf6db572cd52fac889712ed738680fc0"
server_key_version: "encryptionkey@0"
I20260924 16:11:45.682825 26405 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 16:11:45.687530 26420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:45.689586 26405 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:11:45.689842 26405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "971fcac06fc145858d467b5026ffe6a8"
format_stamp: "Formatted at 2026-09-24 16:11:45 on dist-test-slave-1gc3"
server_key: "9be8397596a29520cefc0fcd2595938e"
server_key_iv: "bf6db572cd52fac889712ed738680fc0"
server_key_version: "encryptionkey@0"
I20260924 16:11:45.690078 26405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:45.710079 26405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:45.714906 26405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:45.715196 26405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:45.736740 26405 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:40689
I20260924 16:11:45.736815 26471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:40689 every 8 connection(s)
I20260924 16:11:45.738279 26405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:45.742753 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 26405
I20260924 16:11:45.742971 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:11:45.743700 3931 external_mini_cluster.cc:1428] Setting key b1c2135fbc88bf0ae4d625e70fbfb9a4
I20260924 16:11:45.743753 26472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:45.761018 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:45.746001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51599 (local address 127.3.214.254:40689)
0924 16:11:45.746940 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:45.746954 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:45.747017 (+ 63us) server_negotiation.cc:407] Connection header received
0924 16:11:45.749574 (+ 2557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:45.749602 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:45.750147 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:45.750577 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:45.751397 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:45.752911 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:45.755272 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:45.756172 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:45.756380 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:45.758163 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:45.758175 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:45.758200 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:45.758866 (+ 666us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:45.758978 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:45.759009 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:45.759133 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:45.759464 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 16:11:45.759792 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":409,"threads_started":1}
I20260924 16:11:45.761857 26472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 971fcac06fc145858d467b5026ffe6a8: Bootstrap starting.
I20260924 16:11:45.766551 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:11:45.767168 26472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 971fcac06fc145858d467b5026ffe6a8: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:45.768977 26472 log.cc:824] T 00000000000000000000000000000000 P 971fcac06fc145858d467b5026ffe6a8: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:45.774053 26472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 971fcac06fc145858d467b5026ffe6a8: No bootstrap required, opened a new log
I20260924 16:11:45.781311 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:45.773231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44021 (local address 127.3.214.254:40689)
0924 16:11:45.773462 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:45.773468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:45.773974 (+ 506us) server_negotiation.cc:407] Connection header received
0924 16:11:45.774321 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:45.774326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:45.774405 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:45.774508 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:45.775296 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:45.776045 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:45.778084 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:45.778663 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:45.778778 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:45.780374 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:45.780383 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:45.780387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:45.780411 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:45.780465 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:45.780493 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:45.780585 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:45.780774 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 16:11:45.780921 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 16:11:45.785729 26472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 971fcac06fc145858d467b5026ffe6a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971fcac06fc145858d467b5026ffe6a8" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40689 } }
I20260924 16:11:45.786487 26472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 971fcac06fc145858d467b5026ffe6a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:45.786628 26472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 971fcac06fc145858d467b5026ffe6a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971fcac06fc145858d467b5026ffe6a8, State: Initialized, Role: FOLLOWER
I20260924 16:11:45.787328 26472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 971fcac06fc145858d467b5026ffe6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971fcac06fc145858d467b5026ffe6a8" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40689 } }
I20260924 16:11:45.787588 26472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 971fcac06fc145858d467b5026ffe6a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:45.787710 26472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 971fcac06fc145858d467b5026ffe6a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:45.787866 26472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 971fcac06fc145858d467b5026ffe6a8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:45.790584 26472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 971fcac06fc145858d467b5026ffe6a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971fcac06fc145858d467b5026ffe6a8" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40689 } }
I20260924 16:11:45.791296 26472 leader_election.cc:302] T 00000000000000000000000000000000 P 971fcac06fc145858d467b5026ffe6a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 971fcac06fc145858d467b5026ffe6a8; no voters:
I20260924 16:11:45.792066 26472 leader_election.cc:288] T 00000000000000000000000000000000 P 971fcac06fc145858d467b5026ffe6a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:45.792287 26482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 971fcac06fc145858d467b5026ffe6a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:45.793637 26482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 971fcac06fc145858d467b5026ffe6a8 [term 1 LEADER]: Becoming Leader. State: Replica: 971fcac06fc145858d467b5026ffe6a8, State: Running, Role: LEADER
I20260924 16:11:45.794548 26482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 971fcac06fc145858d467b5026ffe6a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971fcac06fc145858d467b5026ffe6a8" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40689 } }
I20260924 16:11:45.795056 26472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 971fcac06fc145858d467b5026ffe6a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:45.798028 26483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 971fcac06fc145858d467b5026ffe6a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "971fcac06fc145858d467b5026ffe6a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971fcac06fc145858d467b5026ffe6a8" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40689 } } }
I20260924 16:11:45.798277 26484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 971fcac06fc145858d467b5026ffe6a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 971fcac06fc145858d467b5026ffe6a8. Latest consensus state: current_term: 1 leader_uuid: "971fcac06fc145858d467b5026ffe6a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971fcac06fc145858d467b5026ffe6a8" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 40689 } } }
I20260924 16:11:45.798693 26483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 971fcac06fc145858d467b5026ffe6a8 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:45.798734 26484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 971fcac06fc145858d467b5026ffe6a8 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:45.800796 26491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:45.807683 26491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:45.810369 26499 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 16:11:45.820611 26491 catalog_manager.cc:1347] Generated new cluster ID: 9a57694db2ba49adbe8206d0265794d9
I20260924 16:11:45.820780 26491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:45.847452 26491 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:45.849155 26491 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:45.856216 26491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 971fcac06fc145858d467b5026ffe6a8: Generated new TSK 0
I20260924 16:11:45.857453 26491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:11:45.874459 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 26405
W20260924 16:11:45.876237 26467 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51599
W20260924 16:11:45.877761 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 26405: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:11:45.877974 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 26405
2026-09-24T16:11:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2048 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:45 dist-test-slave-1gc3 krb5kdc[26503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:45 dist-test-slave-1gc3 krb5kdc[26503](info): set up 2 sockets
Sep 24 16:11:45 dist-test-slave-1gc3 krb5kdc[26503](info): commencing operation
krb5kdc: starting...
W20260924 16:11:46.948215 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:11:47 dist-test-slave-1gc3 krb5kdc[26503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:11:47.282332 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:34271
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:34999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:34271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:47.714797 26516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:47.715232 26516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:47.715351 26516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:47.723974 26516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:47.724152 26516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:47.724203 26516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:47.724282 26516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:47.724334 26516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:47.736160 26516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:34999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:34271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:34271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:47.738026 26516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:47.739678 26516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:47.754674 26522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:47.755012 26521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:47.754993 26524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:47.756043 26516 server_base.cc:1034] running on GCE node
I20260924 16:11:47.757164 26516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:47.759199 26516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:47.760468 26516 hybrid_clock.cc:648] HybridClock initialized: now 1790266307760409 us; error 48 us; skew 500 ppm
I20260924 16:11:47.760901 26516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:47 dist-test-slave-1gc3 krb5kdc[26503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266307, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:47.768864 26516 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:11:47.771085 26516 webserver.cc:460] Webserver started at http://127.3.214.254:44589/ using document root <none> and password file <none>
I20260924 16:11:47.771996 26516 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:47.772130 26516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:47.772504 26516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:47.776333 26516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "6531c1c691bf45119acce84aea1fc4a5"
format_stamp: "Formatted at 2026-09-24 16:11:47 on dist-test-slave-1gc3"
server_key: "83e2c7edf019c80a0a4ee3c98d61e102"
server_key_iv: "5bf1c942d04ee69da63b81f284aa8ec6"
server_key_version: "encryptionkey@0"
I20260924 16:11:47.777321 26516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "6531c1c691bf45119acce84aea1fc4a5"
format_stamp: "Formatted at 2026-09-24 16:11:47 on dist-test-slave-1gc3"
server_key: "83e2c7edf019c80a0a4ee3c98d61e102"
server_key_iv: "5bf1c942d04ee69da63b81f284aa8ec6"
server_key_version: "encryptionkey@0"
I20260924 16:11:47.784220 26516 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260924 16:11:47.789281 26531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:47.791399 26516 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:11:47.791642 26516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "6531c1c691bf45119acce84aea1fc4a5"
format_stamp: "Formatted at 2026-09-24 16:11:47 on dist-test-slave-1gc3"
server_key: "83e2c7edf019c80a0a4ee3c98d61e102"
server_key_iv: "5bf1c942d04ee69da63b81f284aa8ec6"
server_key_version: "encryptionkey@0"
I20260924 16:11:47.791903 26516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:47.799324 26516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:47.804872 26516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:47.805229 26516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:47.828759 26516 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:34271
I20260924 16:11:47.828821 26582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:34271 every 8 connection(s)
I20260924 16:11:47.830709 26516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:47.835314 26583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:47.839231 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 26516
I20260924 16:11:47.839448 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:11:47.840065 3931 external_mini_cluster.cc:1428] Setting key a9c8edc7da33e2202064c9e3a74bcb28
I20260924 16:11:47.850548 26586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.842724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60407 (local address 127.3.214.254:34271)
0924 16:11:47.843777 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:47.843794 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:47.843856 (+ 62us) server_negotiation.cc:407] Connection header received
0924 16:11:47.845922 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:47.845957 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:47.846575 (+ 618us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:47.847076 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:47.848408 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:47.848416 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:47.848433 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:47.848452 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:47.848528 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:47.848555 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:47.848794 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:47.848915 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 16:11:47.849220 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":461,"threads_started":1}
I20260924 16:11:47.851015 26583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6531c1c691bf45119acce84aea1fc4a5: Bootstrap starting.
I20260924 16:11:47.855031 26583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6531c1c691bf45119acce84aea1fc4a5: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:47.855396 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:11:47.856491 26583 log.cc:824] T 00000000000000000000000000000000 P 6531c1c691bf45119acce84aea1fc4a5: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:47.860756 26583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6531c1c691bf45119acce84aea1fc4a5: No bootstrap required, opened a new log
I20260924 16:11:47.864120 26586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.861013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40291 (local address 127.3.214.254:34271)
0924 16:11:47.861247 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:47.861255 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:47.861816 (+ 561us) server_negotiation.cc:407] Connection header received
0924 16:11:47.862000 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:47.862005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:47.862145 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:47.862299 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:47.863289 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:47.863295 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:47.863299 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:47.863324 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:47.863383 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:47.863405 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:47.863578 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:47.863728 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 16:11:47.863796 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 16:11:47.869803 26583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6531c1c691bf45119acce84aea1fc4a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6531c1c691bf45119acce84aea1fc4a5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34271 } }
I20260924 16:11:47.870282 26583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6531c1c691bf45119acce84aea1fc4a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:47.870365 26583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6531c1c691bf45119acce84aea1fc4a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6531c1c691bf45119acce84aea1fc4a5, State: Initialized, Role: FOLLOWER
I20260924 16:11:47.870936 26583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6531c1c691bf45119acce84aea1fc4a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6531c1c691bf45119acce84aea1fc4a5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34271 } }
I20260924 16:11:47.871165 26583 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6531c1c691bf45119acce84aea1fc4a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:47.871277 26583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6531c1c691bf45119acce84aea1fc4a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:47.871419 26583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6531c1c691bf45119acce84aea1fc4a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:47.873642 26583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6531c1c691bf45119acce84aea1fc4a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6531c1c691bf45119acce84aea1fc4a5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34271 } }
I20260924 16:11:47.874248 26583 leader_election.cc:302] T 00000000000000000000000000000000 P 6531c1c691bf45119acce84aea1fc4a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6531c1c691bf45119acce84aea1fc4a5; no voters:
I20260924 16:11:47.874927 26583 leader_election.cc:288] T 00000000000000000000000000000000 P 6531c1c691bf45119acce84aea1fc4a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:47.875201 26593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6531c1c691bf45119acce84aea1fc4a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:47.876624 26593 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6531c1c691bf45119acce84aea1fc4a5 [term 1 LEADER]: Becoming Leader. State: Replica: 6531c1c691bf45119acce84aea1fc4a5, State: Running, Role: LEADER
I20260924 16:11:47.877369 26583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6531c1c691bf45119acce84aea1fc4a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:47.877342 26593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6531c1c691bf45119acce84aea1fc4a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6531c1c691bf45119acce84aea1fc4a5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34271 } }
I20260924 16:11:47.881072 26595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6531c1c691bf45119acce84aea1fc4a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6531c1c691bf45119acce84aea1fc4a5. Latest consensus state: current_term: 1 leader_uuid: "6531c1c691bf45119acce84aea1fc4a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6531c1c691bf45119acce84aea1fc4a5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34271 } } }
I20260924 16:11:47.881628 26595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6531c1c691bf45119acce84aea1fc4a5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:47.881968 26594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6531c1c691bf45119acce84aea1fc4a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6531c1c691bf45119acce84aea1fc4a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6531c1c691bf45119acce84aea1fc4a5" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34271 } } }
I20260924 16:11:47.882426 26594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6531c1c691bf45119acce84aea1fc4a5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:47.887545 26602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:47.895059 26602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:47.908097 26602 catalog_manager.cc:1347] Generated new cluster ID: 396b049fddd04f42bb762ef07dbcfbc8
I20260924 16:11:47.908244 26602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:47.927428 26602 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:47.927934 26602 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:47.943701 26602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6531c1c691bf45119acce84aea1fc4a5: Generated new TSK 0
I20260924 16:11:47.945396 26602 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:11:47.957544 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 26516
W20260924 16:11:47.958999 26578 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60407
W20260924 16:11:47.960044 3931 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 26516: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:11:47.960209 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 26516
2026-09-24T16:11:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2084 ms)
[----------] 18 tests from AuthTokenIssuingTest (37845 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:48 dist-test-slave-1gc3 krb5kdc[26614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:48 dist-test-slave-1gc3 krb5kdc[26614](info): set up 2 sockets
Sep 24 16:11:48 dist-test-slave-1gc3 krb5kdc[26614](info): commencing operation
krb5kdc: starting...
W20260924 16:11:49.036644 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.003s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:11:49 dist-test-slave-1gc3 krb5kdc[26614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:11:49.384590 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:34619
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:45347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:34619,127.3.214.253:36527,127.3.214.252:45929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:49.818542 26627 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:11:49.819022 26627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:49.819146 26627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:49.819295 26627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:49.827957 26627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:49.828097 26627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:49.828164 26627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:49.828258 26627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:49.828351 26627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:49.840329 26627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:45347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:34619,127.3.214.253:36527,127.3.214.252:45929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:34619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:49.842392 26627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:49.844333 26627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:49.859503 26632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:49.859478 26635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:49.859525 26633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:49.860755 26627 server_base.cc:1034] running on GCE node
I20260924 16:11:49.861836 26627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:49.863852 26627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:49.865118 26627 hybrid_clock.cc:648] HybridClock initialized: now 1790266309865028 us; error 57 us; skew 500 ppm
I20260924 16:11:49.865540 26627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:49 dist-test-slave-1gc3 krb5kdc[26614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:49.869572 26627 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:11:49.871788 26627 webserver.cc:460] Webserver started at http://127.3.214.254:39535/ using document root <none> and password file <none>
I20260924 16:11:49.872771 26627 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:49.872898 26627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:49.873281 26627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:49.877285 26627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "5c8959891c254e6180c804046d250041"
format_stamp: "Formatted at 2026-09-24 16:11:49 on dist-test-slave-1gc3"
server_key: "2907a7874da841e8106536613972f512"
server_key_iv: "5d5a92667eadd09af175841163b3a8c7"
server_key_version: "encryptionkey@0"
I20260924 16:11:49.878407 26627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "5c8959891c254e6180c804046d250041"
format_stamp: "Formatted at 2026-09-24 16:11:49 on dist-test-slave-1gc3"
server_key: "2907a7874da841e8106536613972f512"
server_key_iv: "5d5a92667eadd09af175841163b3a8c7"
server_key_version: "encryptionkey@0"
I20260924 16:11:49.886616 26627 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260924 16:11:49.892745 26642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:49.895098 26627 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260924 16:11:49.895401 26627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "5c8959891c254e6180c804046d250041"
format_stamp: "Formatted at 2026-09-24 16:11:49 on dist-test-slave-1gc3"
server_key: "2907a7874da841e8106536613972f512"
server_key_iv: "5d5a92667eadd09af175841163b3a8c7"
server_key_version: "encryptionkey@0"
I20260924 16:11:49.895717 26627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:49.915210 26627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:49.921271 26627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:49.921618 26627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:49.945192 26627 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:34619
I20260924 16:11:49.945235 26693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:34619 every 8 connection(s)
I20260924 16:11:49.947038 26627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:49.951694 26694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:49.952296 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 26627
I20260924 16:11:49.952533 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:11:49.953382 3931 external_mini_cluster.cc:1428] Setting key 032d8dad67826bc23a4f1c4b1358df38
I20260924 16:11:49.961663 26694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34619 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:11:49 dist-test-slave-1gc3 krb5kdc[26614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.214.253@KRBTEST.COM; defaulting to no policy
I20260924 16:11:49.994151 26697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:49.966521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43664 (local address 127.3.214.254:34619)
0924 16:11:49.968921 (+ 2400us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:49.968934 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:49.969007 (+ 73us) server_negotiation.cc:407] Connection header received
0924 16:11:49.971469 (+ 2462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:49.971479 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:49.971838 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:49.972082 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:49.974386 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:49.975815 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:49.978640 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:49.979513 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:49.979719 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:49.984821 (+ 5102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:49.984834 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:49.984854 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:49.984938 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:49.988181 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:49.989529 (+ 1348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:49.989539 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:49.989542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:49.989782 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:49.990428 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:49.990433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:49.990435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:49.991123 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:49.991321 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:49.992261 (+ 940us) server_negotiation.cc:299] Negotiation successful
0924 16:11:49.992942 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":360,"server-negotiator.queue_time_us":2122,"spinlock_wait_cycles":22784,"thread_start_us":1845,"threads_started":1}
I20260924 16:11:49.994323 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:49.966520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:34619 (local address 127.0.0.1:43664)
0924 16:11:49.967683 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:49.967760 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:49.970771 (+ 3011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:49.972208 (+ 1437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:49.972218 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:49.973206 (+ 988us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:49.973878 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:49.973917 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:49.976189 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:49.976196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:49.978318 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:49.978348 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:49.979915 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:49.979923 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:49.980144 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:49.981049 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:49.981085 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:49.984354 (+ 3269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:49.988418 (+ 4064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:49.988442 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:49.988463 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:49.989257 (+ 794us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:49.989939 (+ 682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:49.989944 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:49.989948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:49.990257 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:49.991301 (+ 1044us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:49.991312 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:49.991657 (+ 345us) client_negotiation.cc:769] Sending connection context
0924 16:11:49.992038 (+ 381us) client_negotiation.cc:240] Negotiation successful
0924 16:11:49.992694 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":1,"thread_start_us":485,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.253@KRBTEST.COM" created.
I20260924 16:11:49.998140 26694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:11:49.999688 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:49.999145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:36527 (local address 127.0.0.1:38330)
0924 16:11:49.999364 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:49.999517 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260924 16:11:50.000124 26643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.214.253:36527: connect: Connection refused (error 111)
I20260924 16:11:50.002470 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.001879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:36527 (local address 127.0.0.1:38338)
0924 16:11:50.002175 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.002269 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
W20260924 16:11:50.002710 26694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.253:36527: Network error: Client connection negotiation failed: client connection to 127.3.214.253:36527: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.253.keytab.
Entry for principal kudu/127.3.214.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.253.keytab.
I20260924 16:11:50.034608 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.253:36527
--webserver_interface=127.3.214.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.253.keytab
--principal=kudu/127.3.214.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:45347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:34619,127.3.214.253:36527,127.3.214.252:45929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:11:50.052368 26694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } attempt: 1
I20260924 16:11:50.054258 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.053541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:36527 (local address 127.0.0.1:38342)
0924 16:11:50.053821 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.054007 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260924 16:11:50.055810 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.055306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:36527 (local address 127.0.0.1:38354)
0924 16:11:50.055501 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.055603 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 16:11:50.056073 26694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.253:36527: Network error: Client connection negotiation failed: client connection to 127.3.214.253:36527: connect: Connection refused (error 111)
I20260924 16:11:50.124437 26694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } attempt: 2
I20260924 16:11:50.126343 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.125461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:36527 (local address 127.0.0.1:38370)
0924 16:11:50.125917 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.126094 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260924 16:11:50.128116 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.127453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:36527 (local address 127.0.0.1:38374)
0924 16:11:50.127775 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.127901 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260924 16:11:50.128515 26694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.253:36527: Network error: Client connection negotiation failed: client connection to 127.3.214.253:36527: connect: Connection refused (error 111)
I20260924 16:11:50.219889 26694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } attempt: 3
I20260924 16:11:50.221566 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.220806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:36527 (local address 127.0.0.1:38386)
0924 16:11:50.221168 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.221354 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260924 16:11:50.222975 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.222480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:36527 (local address 127.0.0.1:38388)
0924 16:11:50.222655 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.222774 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 16:11:50.223299 26694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.253:36527: Network error: Client connection negotiation failed: client connection to 127.3.214.253:36527: connect: Connection refused (error 111)
I20260924 16:11:50.366652 26694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } attempt: 4
I20260924 16:11:50.368415 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.367586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:36527 (local address 127.0.0.1:38402)
0924 16:11:50.368094 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.368228 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264}
I20260924 16:11:50.370011 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.369512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:36527 (local address 127.0.0.1:38414)
0924 16:11:50.369710 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.369839 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260924 16:11:50.370349 26694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.253:36527: Network error: Client connection negotiation failed: client connection to 127.3.214.253:36527: connect: Connection refused (error 111)
W20260924 16:11:50.431962 26701 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:11:50.432338 26701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:50.432449 26701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:50.432587 26701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:50.440574 26701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:50.440711 26701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:50.440773 26701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:50.440896 26701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:50.440994 26701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:50.452059 26701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:45347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:34619,127.3.214.253:36527,127.3.214.252:45929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.253.keytab
--principal=kudu/127.3.214.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.253:36527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.214.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:50.453621 26701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:50.455138 26701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:50.468724 26706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:50.468791 26707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:50.469177 26709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:50.470254 26701 server_base.cc:1034] running on GCE node
I20260924 16:11:50.471334 26701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:50.473321 26701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:50.474586 26701 hybrid_clock.cc:648] HybridClock initialized: now 1790266310474481 us; error 65 us; skew 500 ppm
I20260924 16:11:50.475023 26701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:50 dist-test-slave-1gc3 krb5kdc[26614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266310, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:50.479398 26701 init.cc:377] Logged in from keytab as kudu/127.3.214.253@KRBTEST.COM (short username kudu)
I20260924 16:11:50.481526 26701 webserver.cc:460] Webserver started at http://127.3.214.253:41941/ using document root <none> and password file <none>
I20260924 16:11:50.482583 26701 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:50.482733 26701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:50.483140 26701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:50.487331 26701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-1/data/instance:
uuid: "21f6f1d18fc44164a0434bee5677fcd3"
format_stamp: "Formatted at 2026-09-24 16:11:50 on dist-test-slave-1gc3"
server_key: "86b9e711da04ef806f81fbb9c45bd99f"
server_key_iv: "ad224cf1960d7b3a68a0394a0e3e69af"
server_key_version: "encryptionkey@0"
I20260924 16:11:50.488382 26701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-1/wal/instance:
uuid: "21f6f1d18fc44164a0434bee5677fcd3"
format_stamp: "Formatted at 2026-09-24 16:11:50 on dist-test-slave-1gc3"
server_key: "86b9e711da04ef806f81fbb9c45bd99f"
server_key_iv: "ad224cf1960d7b3a68a0394a0e3e69af"
server_key_version: "encryptionkey@0"
I20260924 16:11:50.495771 26701 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260924 16:11:50.501184 26716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:50.503569 26701 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260924 16:11:50.503898 26701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-1/wal
uuid: "21f6f1d18fc44164a0434bee5677fcd3"
format_stamp: "Formatted at 2026-09-24 16:11:50 on dist-test-slave-1gc3"
server_key: "86b9e711da04ef806f81fbb9c45bd99f"
server_key_iv: "ad224cf1960d7b3a68a0394a0e3e69af"
server_key_version: "encryptionkey@0"
I20260924 16:11:50.504222 26701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:50.522398 26701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:50.528041 26701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:50.528391 26701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:50.551383 26701 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.253:36527
I20260924 16:11:50.551410 26767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.253:36527 every 8 connection(s)
I20260924 16:11:50.553187 26701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-1/data/info.pb
I20260924 16:11:50.557705 26768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:50.559901 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 26701
I20260924 16:11:50.560102 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-1/wal/instance
I20260924 16:11:50.560622 3931 external_mini_cluster.cc:1428] Setting key ac93cd3bf02ec5aa45abd193ee71f3b5
I20260924 16:11:50.566975 26768 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34619 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:11:50 dist-test-slave-1gc3 krb5kdc[26614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266310, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.253@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:11:50.593870 26770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.571038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43670 (local address 127.3.214.254:34619)
0924 16:11:50.571751 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:50.571759 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:50.573257 (+ 1498us) server_negotiation.cc:407] Connection header received
0924 16:11:50.575506 (+ 2249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:50.575514 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:50.575609 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:50.575705 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:50.577508 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:50.578336 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:50.579937 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:50.580380 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:50.580492 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:50.585009 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:50.585015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:50.585019 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:50.585064 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:50.587953 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:50.589407 (+ 1454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:50.589413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:50.589415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:50.589571 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:50.590358 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:50.590398 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:50.590402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:50.590830 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:50.591032 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:50.592572 (+ 1540us) server_negotiation.cc:299] Negotiation successful
0924 16:11:50.593015 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":333,"threads_started":1}
WARNING: no policy specified for kudu/127.3.214.252@KRBTEST.COM; defaulting to no policy
I20260924 16:11:50.595149 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.571915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:34619 (local address 127.0.0.1:43670)
0924 16:11:50.573024 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.573086 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:50.575147 (+ 2061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:50.576019 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:50.576029 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:50.576641 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:50.577275 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:50.577293 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:50.578507 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:50.578511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:50.579783 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:50.579797 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:50.580628 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:50.580636 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:50.580863 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:50.581671 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:50.581695 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:50.584616 (+ 2921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:50.588228 (+ 3612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:50.588245 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:50.588274 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:50.589209 (+ 935us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:50.589727 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:50.589733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:50.589738 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:50.590171 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:50.591072 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:50.591087 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:50.591871 (+ 784us) client_negotiation.cc:769] Sending connection context
0924 16:11:50.592318 (+ 447us) client_negotiation.cc:240] Negotiation successful
0924 16:11:50.593140 (+ 822us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":427,"threads_started":1}
I20260924 16:11:50.598011 26768 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.252@KRBTEST.COM" created.
Sep 24 16:11:50 dist-test-slave-1gc3 krb5kdc[26614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266310, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.253@KRBTEST.COM for kudu/127.3.214.253@KRBTEST.COM
I20260924 16:11:50.623225 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.602662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:36527 (local address 127.0.0.1:38424)
0924 16:11:50.603036 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.603065 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:50.603274 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:50.604421 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:50.604426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:50.605045 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:50.605294 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:50.605308 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:50.607421 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:50.607426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:50.608856 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:50.608876 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:50.611116 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:50.611122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:50.611326 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:50.612156 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:50.612182 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:50.615035 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:50.619769 (+ 4734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:50.619777 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:50.619781 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:50.620484 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:50.621033 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:50.621038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:50.621042 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:50.621259 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:50.622050 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:50.622055 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:50.622290 (+ 235us) client_negotiation.cc:769] Sending connection context
0924 16:11:50.622458 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 16:11:50.622789 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260924 16:11:50.623386 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.602999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38424 (local address 127.3.214.253:36527)
0924 16:11:50.603836 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:50.603844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:50.603879 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:11:50.603955 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:50.603962 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:50.604207 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:50.604361 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:50.605544 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:50.607218 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:50.609157 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:50.610602 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:50.610787 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:50.615397 (+ 4610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:50.615421 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:50.615446 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:50.615489 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:50.619488 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:50.620703 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:50.620713 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:50.620715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:50.620882 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:50.621388 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:50.621392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:50.621394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:50.621874 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:50.622085 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:50.622642 (+ 557us) server_negotiation.cc:299] Negotiation successful
0924 16:11:50.622930 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":445,"threads_started":1}
I20260924 16:11:50.626756 26768 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:11:50.628234 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.627612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:45929 (local address 127.0.0.1:51152)
0924 16:11:50.627824 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.627980 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260924 16:11:50.628564 26720 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
I20260924 16:11:50.630726 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.630200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:45929 (local address 127.0.0.1:51166)
0924 16:11:50.630437 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.630557 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260924 16:11:50.631022 26768 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:45929: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.252.keytab.
Entry for principal kudu/127.3.214.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.252.keytab.
I20260924 16:11:50.646179 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.252:45929
--webserver_interface=127.3.214.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.252.keytab
--principal=kudu/127.3.214.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:45347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:34619,127.3.214.253:36527,127.3.214.252:45929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:11:50.669642 26694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } attempt: 5
Sep 24 16:11:50 dist-test-slave-1gc3 krb5kdc[26614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for kudu/127.3.214.253@KRBTEST.COM
I20260924 16:11:50.680472 26768 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } attempt: 1
I20260924 16:11:50.681970 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.681398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:45929 (local address 127.0.0.1:51172)
0924 16:11:50.681664 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.681786 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 16:11:50.683468 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.682866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:45929 (local address 127.0.0.1:51186)
0924 16:11:50.683124 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.683256 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260924 16:11:50.683862 26768 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:45929: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
I20260924 16:11:50.687750 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.670619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:36527 (local address 127.0.0.1:38440)
0924 16:11:50.670913 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.670939 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:50.671105 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:50.671673 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:50.671678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:50.672219 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:50.672496 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:50.672511 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:50.673688 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:50.673694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:50.675173 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:50.675191 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:50.676240 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:50.676247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:50.676493 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:50.677235 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:50.677260 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:50.680082 (+ 2822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:50.684883 (+ 4801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:50.684890 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:50.684895 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:50.685506 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:50.685995 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:50.686001 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:50.686005 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:50.686209 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:50.686928 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:50.686933 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:50.687108 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 16:11:50.687230 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 16:11:50.687494 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260924 16:11:50.688040 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.670913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38440 (local address 127.3.214.253:36527)
0924 16:11:50.671222 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:50.671249 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:50.671277 (+ 28us) server_negotiation.cc:407] Connection header received
0924 16:11:50.671357 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:50.671362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:50.671473 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:50.671586 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:50.672710 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:50.673528 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:50.675397 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:50.675992 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:50.676180 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:50.680461 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:50.680471 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:50.680477 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:50.680542 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:50.684686 (+ 4144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:50.685699 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:50.685704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:50.685707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:50.685880 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:50.686334 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:50.686339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:50.686342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:50.686793 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:50.686962 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:50.687394 (+ 432us) server_negotiation.cc:299] Negotiation successful
0924 16:11:50.687690 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:11:50.689312 26694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:11:50.690871 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.690272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:45929 (local address 127.0.0.1:51188)
0924 16:11:50.690501 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.690639 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 16:11:50.692098 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.691648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:45929 (local address 127.0.0.1:51200)
0924 16:11:50.691815 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.691942 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 16:11:50.692364 26694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:45929: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
I20260924 16:11:50.743698 26694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } attempt: 1
I20260924 16:11:50.745420 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.744682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:45929 (local address 127.0.0.1:51214)
0924 16:11:50.745003 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.745162 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260924 16:11:50.746819 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.746368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:45929 (local address 127.0.0.1:51220)
0924 16:11:50.746544 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.746664 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 16:11:50.747097 26694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:45929: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
I20260924 16:11:50.752134 26768 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } attempt: 2
I20260924 16:11:50.753815 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.753203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:45929 (local address 127.0.0.1:51226)
0924 16:11:50.753496 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.753620 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260924 16:11:50.754908 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.754591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:45929 (local address 127.0.0.1:51230)
0924 16:11:50.754717 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.754797 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 16:11:50.755128 26768 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:45929: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
I20260924 16:11:50.815393 26694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } attempt: 2
I20260924 16:11:50.817163 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.816391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:45929 (local address 127.0.0.1:51242)
0924 16:11:50.816728 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.816886 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260924 16:11:50.818607 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.818084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:45929 (local address 127.0.0.1:51244)
0924 16:11:50.818308 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.818420 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 16:11:50.818909 26694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:45929: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
I20260924 16:11:50.846382 26768 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } attempt: 3
I20260924 16:11:50.847898 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.847268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:45929 (local address 127.0.0.1:51256)
0924 16:11:50.847563 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.847704 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260924 16:11:50.849054 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.848578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:45929 (local address 127.0.0.1:51270)
0924 16:11:50.848854 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.848938 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260924 16:11:50.849267 26768 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:45929: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
I20260924 16:11:50.925256 26694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } attempt: 3
I20260924 16:11:50.926980 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.926228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:45929 (local address 127.0.0.1:51276)
0924 16:11:50.926523 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.926712 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 16:11:50.928447 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.927845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:45929 (local address 127.0.0.1:51284)
0924 16:11:50.928105 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.928254 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260924 16:11:50.928767 26694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:45929: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
I20260924 16:11:50.992564 26768 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } attempt: 4
I20260924 16:11:50.993844 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.993330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:45929 (local address 127.0.0.1:51294)
0924 16:11:50.993585 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.993675 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260924 16:11:50.995260 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.994893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:45929 (local address 127.0.0.1:51304)
0924 16:11:50.995063 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.995149 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 16:11:50.995507 26768 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:45929: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
W20260924 16:11:51.084065 26776 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:11:51.084547 26776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:51.084630 26776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:51.084725 26776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:51.093258 26776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:51.093411 26776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:51.093477 26776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:51.093526 26776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:51.093609 26776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:51.105779 26776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:45347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:34619,127.3.214.253:36527,127.3.214.252:45929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.252.keytab
--principal=kudu/127.3.214.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.252:45929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.214.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:51.106037 26694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } attempt: 4
I20260924 16:11:51.107527 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:51.106910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:45929 (local address 127.0.0.1:51312)
0924 16:11:51.107190 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:51.107328 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260924 16:11:51.107990 26776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:51.108916 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:51.108454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:45929 (local address 127.0.0.1:51320)
0924 16:11:51.108629 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:51.108722 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 16:11:51.109275 26694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:45929: Network error: Client connection negotiation failed: client connection to 127.3.214.252:45929: connect: Connection refused (error 111)
I20260924 16:11:51.109977 26776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:51.124642 26782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:51.124820 26781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:51.124703 26784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:51.126281 26776 server_base.cc:1034] running on GCE node
I20260924 16:11:51.127576 26776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:51.130038 26776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:51.131458 26776 hybrid_clock.cc:648] HybridClock initialized: now 1790266311131347 us; error 55 us; skew 500 ppm
I20260924 16:11:51.132011 26776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:51 dist-test-slave-1gc3 krb5kdc[26614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266311, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:51.136850 26776 init.cc:377] Logged in from keytab as kudu/127.3.214.252@KRBTEST.COM (short username kudu)
I20260924 16:11:51.139071 26776 webserver.cc:460] Webserver started at http://127.3.214.252:39005/ using document root <none> and password file <none>
I20260924 16:11:51.140100 26776 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:51.140257 26776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:51.140677 26776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:51.144867 26776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-2/data/instance:
uuid: "5d9cc96eaa4143eb90478e7ba18b2b4f"
format_stamp: "Formatted at 2026-09-24 16:11:51 on dist-test-slave-1gc3"
server_key: "6b3df73aad6816498780f1585e9efd5f"
server_key_iv: "bdeccba2a0a5a06a5c38c6f25c10a860"
server_key_version: "encryptionkey@0"
I20260924 16:11:51.145938 26776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-2/wal/instance:
uuid: "5d9cc96eaa4143eb90478e7ba18b2b4f"
format_stamp: "Formatted at 2026-09-24 16:11:51 on dist-test-slave-1gc3"
server_key: "6b3df73aad6816498780f1585e9efd5f"
server_key_iv: "bdeccba2a0a5a06a5c38c6f25c10a860"
server_key_version: "encryptionkey@0"
I20260924 16:11:51.153753 26776 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20260924 16:11:51.159183 26791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:51.161425 26776 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260924 16:11:51.161705 26776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-2/wal
uuid: "5d9cc96eaa4143eb90478e7ba18b2b4f"
format_stamp: "Formatted at 2026-09-24 16:11:51 on dist-test-slave-1gc3"
server_key: "6b3df73aad6816498780f1585e9efd5f"
server_key_iv: "bdeccba2a0a5a06a5c38c6f25c10a860"
server_key_version: "encryptionkey@0"
I20260924 16:11:51.162012 26776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:51.181329 26776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:51.186985 26776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:51.187326 26776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:51.210513 26842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.252:45929 every 8 connection(s)
I20260924 16:11:51.210554 26776 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.252:45929
I20260924 16:11:51.212585 26776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-2/data/info.pb
I20260924 16:11:51.217252 26843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:51.221915 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 26776
I20260924 16:11:51.222191 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790265920616583-3931-0/minicluster-data/master-2/wal/instance
I20260924 16:11:51.222720 3931 external_mini_cluster.cc:1428] Setting key 4117dd1087423c63adaadb7274b4d775
I20260924 16:11:51.225414 26843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34619 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:11:51 dist-test-slave-1gc3 krb5kdc[26614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266311, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.252@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
Sep 24 16:11:51 dist-test-slave-1gc3 krb5kdc[26614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.252@KRBTEST.COM
I20260924 16:11:51.248740 26847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:51.228740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43680 (local address 127.3.214.254:34619)
0924 16:11:51.229425 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:51.229433 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:51.230286 (+ 853us) server_negotiation.cc:407] Connection header received
0924 16:11:51.230369 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:51.230374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:51.230458 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:51.230553 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:51.232098 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.232876 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.234714 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.235148 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.235282 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:51.239954 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:51.239963 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:51.239967 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:51.240020 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:51.243546 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:51.244789 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:51.244796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:51.244799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:51.244992 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:51.245782 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:51.245789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:51.245792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:51.246299 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:51.246508 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:51.247773 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0924 16:11:51.248253 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":299,"threads_started":1}
I20260924 16:11:51.250273 26846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:51.225199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51326 (local address 127.3.214.252:45929)
0924 16:11:51.226347 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:51.226380 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:51.226445 (+ 65us) server_negotiation.cc:407] Connection header received
0924 16:11:51.228836 (+ 2391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:51.228881 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:51.229669 (+ 788us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:51.230166 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:51.232043 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.233544 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.235159 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.235891 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.236036 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:51.241661 (+ 5625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:51.241675 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:51.241696 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:51.241749 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:51.245073 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:51.246243 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:51.246255 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:51.246259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:51.246450 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:51.247030 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:51.247035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:51.247037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:51.247585 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:51.248136 (+ 551us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:51.248387 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 16:11:51.248718 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":786,"spinlock_wait_cycles":25472,"thread_start_us":466,"threads_started":1}
I20260924 16:11:51.250420 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:51.228719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:34619 (local address 127.0.0.1:43680)
0924 16:11:51.229824 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:51.229892 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:51.230140 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:51.230688 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:51.230696 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:51.231439 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:51.231886 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:51.231909 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.233059 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.233064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:51.234522 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:51.234541 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.235445 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.235459 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:51.235695 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:51.236456 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:51.236483 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:51.239603 (+ 3120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:51.243818 (+ 4215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:51.243843 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:51.243859 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:51.244577 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:51.245213 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:51.245219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:51.245222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:51.245575 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:51.246541 (+ 966us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:51.246555 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:51.247249 (+ 694us) client_negotiation.cc:769] Sending connection context
0924 16:11:51.247573 (+ 324us) client_negotiation.cc:240] Negotiation successful
0924 16:11:51.248349 (+ 776us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":596,"threads_started":1}
I20260924 16:11:51.252909 26843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:11:51 dist-test-slave-1gc3 krb5kdc[26614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
Sep 24 16:11:51 dist-test-slave-1gc3 krb5kdc[26614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266311, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.252@KRBTEST.COM for kudu/127.3.214.253@KRBTEST.COM
I20260924 16:11:51.273017 26847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:51.254539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43686 (local address 127.3.214.254:34619)
0924 16:11:51.254739 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:51.254744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:51.254845 (+ 101us) server_negotiation.cc:407] Connection header received
0924 16:11:51.255083 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:51.255088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:51.255175 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:51.255271 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:51.257132 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.257947 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.259354 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.260042 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.260199 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:51.265472 (+ 5273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:51.265483 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:51.265489 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:51.265549 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:51.269467 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:51.270538 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:51.270545 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:51.270548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:51.270737 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:51.271253 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:51.271258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:51.271260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:51.271625 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:51.271969 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:51.272345 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 16:11:51.272648 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:11:51.274713 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:51.253927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:36527 (local address 127.0.0.1:38454)
0924 16:11:51.254185 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:51.254207 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:51.254401 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:51.259031 (+ 4630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:51.259038 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:51.259736 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:51.260002 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:51.260018 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.261611 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.261616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:51.262745 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:51.262767 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.263646 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.263661 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:51.263854 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:51.264785 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:51.264811 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:51.267952 (+ 3141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:51.271636 (+ 3684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:51.271642 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:51.271646 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:51.272101 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:51.272579 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:51.272595 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:51.272598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:51.272824 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:51.273686 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:51.273692 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:51.273874 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 16:11:51.274007 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 16:11:51.274369 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 16:11:51.274986 26849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:51.254303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38454 (local address 127.3.214.253:36527)
0924 16:11:51.258637 (+ 4334us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:51.258642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:51.258662 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:11:51.258725 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:51.258730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:51.258804 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:51.258922 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:51.260506 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.261418 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.262957 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.263439 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.263566 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:51.268271 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:51.268299 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:51.268303 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:51.268349 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:51.271435 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:51.272252 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:51.272257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:51.272259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:51.272406 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:51.273114 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:51.273121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:51.273124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:51.273524 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:51.273710 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:51.274214 (+ 504us) server_negotiation.cc:299] Negotiation successful
0924 16:11:51.274529 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4194,"thread_start_us":4047,"threads_started":1}
I20260924 16:11:51.276130 26843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:11:51 dist-test-slave-1gc3 krb5kdc[26614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266311, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.252@KRBTEST.COM for kudu/127.3.214.252@KRBTEST.COM
Sep 24 16:11:51 dist-test-slave-1gc3 krb5kdc[26614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.253@KRBTEST.COM
I20260924 16:11:51.292660 26846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:51.277205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51338 (local address 127.3.214.252:45929)
0924 16:11:51.277355 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:51.277360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:51.277377 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:11:51.277600 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:51.277604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:51.277678 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:51.277758 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:51.278667 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.279404 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.280600 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.281017 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.281131 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:51.284784 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:51.284791 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:51.284795 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:51.284844 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:51.288248 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:51.289329 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:51.289333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:51.289336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:51.289531 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:51.290072 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:51.290079 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:51.290082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:51.290638 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:51.290851 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:51.291486 (+ 635us) server_negotiation.cc:299] Negotiation successful
0924 16:11:51.291979 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":65}
I20260924 16:11:51.292711 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:51.277045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:45929 (local address 127.0.0.1:51338)
0924 16:11:51.277270 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:51.277292 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:51.277480 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:51.277794 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:51.277798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:51.278341 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:51.278525 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:51.278536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.279536 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.279553 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:51.280465 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:51.280477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.281165 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.281172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:51.281391 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:51.282039 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:51.282058 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:51.284459 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:51.288511 (+ 4052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:51.288519 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:51.288523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:51.289165 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:51.289681 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:51.289686 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:51.289689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:51.289894 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:51.290866 (+ 972us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:51.290870 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:51.291107 (+ 237us) client_negotiation.cc:769] Sending connection context
0924 16:11:51.291322 (+ 215us) client_negotiation.cc:240] Negotiation successful
0924 16:11:51.291720 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 16:11:51.294747 26768 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } attempt: 5
I20260924 16:11:51.296576 26849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:51.275638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38458 (local address 127.3.214.253:36527)
0924 16:11:51.275822 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:51.275828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:51.275848 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:11:51.276179 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:51.276186 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:51.276281 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:51.276378 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:51.278273 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.279077 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.280495 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.280933 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.281058 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:51.286394 (+ 5336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:51.286401 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:51.286405 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:51.286453 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:51.289672 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:51.291089 (+ 1417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:51.291096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:51.291099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:51.291315 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:51.291973 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:51.291978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:51.291981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:51.293279 (+ 1298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:51.293674 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:51.294439 (+ 765us) server_negotiation.cc:299] Negotiation successful
0924 16:11:51.294925 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:11:51.301570 26843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d9cc96eaa4143eb90478e7ba18b2b4f: Bootstrap starting.
Sep 24 16:11:51 dist-test-slave-1gc3 krb5kdc[26614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266310, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.253@KRBTEST.COM for kudu/127.3.214.252@KRBTEST.COM
I20260924 16:11:51.305790 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:11:51.306289 26843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d9cc96eaa4143eb90478e7ba18b2b4f: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:51.307999 26843 log.cc:824] T 00000000000000000000000000000000 P 5d9cc96eaa4143eb90478e7ba18b2b4f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:51.313201 26843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d9cc96eaa4143eb90478e7ba18b2b4f: No bootstrap required, opened a new log
I20260924 16:11:51.309751 26669 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5c8959891c254e6180c804046d250041"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43686
I20260924 16:11:51.314493 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:51.295546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:45929 (local address 127.0.0.1:51352)
0924 16:11:51.295756 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:51.295779 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:51.295913 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:51.296286 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:51.296290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:51.296800 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:51.297059 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:51.297073 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.298750 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.298757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:51.300057 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:51.300122 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.301429 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.301437 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:51.301745 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:51.302565 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:51.302588 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:51.305626 (+ 3038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:51.310434 (+ 4808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:51.310442 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:51.310447 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:51.311186 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:51.311938 (+ 752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:51.311944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:51.311948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:51.312190 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:51.313123 (+ 933us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:51.313128 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:51.313360 (+ 232us) client_negotiation.cc:769] Sending connection context
0924 16:11:51.313553 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 16:11:51.313990 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 16:11:51.314721 26846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:51.295707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51352 (local address 127.3.214.252:45929)
0924 16:11:51.295954 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:51.295961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:51.295981 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:11:51.296047 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:51.296053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:51.296145 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:51.296235 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:51.297276 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.298542 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.300429 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.301218 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.301362 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:51.305955 (+ 4593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:51.305963 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:51.305968 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:51.306015 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:51.310174 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:51.311451 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:51.311460 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:51.311464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:51.311735 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:51.312346 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:51.312352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:51.312356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:51.312938 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:51.313133 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:51.313936 (+ 803us) server_negotiation.cc:299] Negotiation successful
0924 16:11:51.314291 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260924 16:11:51.321655 26768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21f6f1d18fc44164a0434bee5677fcd3: Bootstrap starting.
I20260924 16:11:51.323400 26843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d9cc96eaa4143eb90478e7ba18b2b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8959891c254e6180c804046d250041" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34619 } } peers { permanent_uuid: "21f6f1d18fc44164a0434bee5677fcd3" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } } peers { permanent_uuid: "5d9cc96eaa4143eb90478e7ba18b2b4f" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } }
I20260924 16:11:51.323945 26843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d9cc96eaa4143eb90478e7ba18b2b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9cc96eaa4143eb90478e7ba18b2b4f, State: Initialized, Role: FOLLOWER
I20260924 16:11:51.324623 26843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d9cc96eaa4143eb90478e7ba18b2b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8959891c254e6180c804046d250041" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34619 } } peers { permanent_uuid: "21f6f1d18fc44164a0434bee5677fcd3" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } } peers { permanent_uuid: "5d9cc96eaa4143eb90478e7ba18b2b4f" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } }
I20260924 16:11:51.326392 26768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21f6f1d18fc44164a0434bee5677fcd3: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:51.326050 26851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9cc96eaa4143eb90478e7ba18b2b4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8959891c254e6180c804046d250041" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34619 } } peers { permanent_uuid: "21f6f1d18fc44164a0434bee5677fcd3" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } } peers { permanent_uuid: "5d9cc96eaa4143eb90478e7ba18b2b4f" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } } }
I20260924 16:11:51.326758 26851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9cc96eaa4143eb90478e7ba18b2b4f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:11:51.327769 26768 log.cc:824] T 00000000000000000000000000000000 P 21f6f1d18fc44164a0434bee5677fcd3: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:51.327848 26843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d9cc96eaa4143eb90478e7ba18b2b4f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:51.331308 26768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21f6f1d18fc44164a0434bee5677fcd3: No bootstrap required, opened a new log
W20260924 16:11:51.341586 26863 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d9cc96eaa4143eb90478e7ba18b2b4f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:11:51.341929 26863 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:11:51.342404 26768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21f6f1d18fc44164a0434bee5677fcd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8959891c254e6180c804046d250041" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34619 } } peers { permanent_uuid: "21f6f1d18fc44164a0434bee5677fcd3" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } } peers { permanent_uuid: "5d9cc96eaa4143eb90478e7ba18b2b4f" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } }
I20260924 16:11:51.342859 26768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21f6f1d18fc44164a0434bee5677fcd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f6f1d18fc44164a0434bee5677fcd3, State: Initialized, Role: FOLLOWER
I20260924 16:11:51.343458 26768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21f6f1d18fc44164a0434bee5677fcd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8959891c254e6180c804046d250041" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34619 } } peers { permanent_uuid: "21f6f1d18fc44164a0434bee5677fcd3" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } } peers { permanent_uuid: "5d9cc96eaa4143eb90478e7ba18b2b4f" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } }
I20260924 16:11:51.344857 26865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21f6f1d18fc44164a0434bee5677fcd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8959891c254e6180c804046d250041" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34619 } } peers { permanent_uuid: "21f6f1d18fc44164a0434bee5677fcd3" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } } peers { permanent_uuid: "5d9cc96eaa4143eb90478e7ba18b2b4f" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } } }
I20260924 16:11:51.345674 26865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21f6f1d18fc44164a0434bee5677fcd3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:11:51.346262 26768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21f6f1d18fc44164a0434bee5677fcd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:51.337823 26776 master_runner.cc:186] Error getting master registration for 127.3.214.254:34619: OK, instance_id { permanent_uuid: "5c8959891c254e6180c804046d250041" instance_seqno: 1790266309938524 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:11:51.350775 26776 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 16:11:51.364409 26876 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 21f6f1d18fc44164a0434bee5677fcd3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:11:51.364702 26876 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:11:51.361913 26701 master_runner.cc:186] Error getting master registration for 127.3.214.254:34619: OK, instance_id { permanent_uuid: "5c8959891c254e6180c804046d250041" instance_seqno: 1790266309938524 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:11:51.373338 26701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:11:51.386547 26694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } attempt: 5
Sep 24 16:11:51 dist-test-slave-1gc3 krb5kdc[26614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266309, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for kudu/127.3.214.252@KRBTEST.COM
I20260924 16:11:51.406428 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:51.387352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:45929 (local address 127.0.0.1:51364)
0924 16:11:51.387654 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:51.387674 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:51.387814 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:51.388206 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:51.388211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:51.388798 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:51.389152 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:51.389168 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.390298 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.390304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:51.391637 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:51.391654 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.392975 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.392982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:51.393203 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:51.393912 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:51.393938 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:51.397447 (+ 3509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:51.402751 (+ 5304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:51.402759 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:51.402764 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:51.403448 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:51.404064 (+ 616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:51.404070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:51.404074 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:51.404292 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:51.405121 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:51.405127 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:51.405317 (+ 190us) client_negotiation.cc:769] Sending connection context
0924 16:11:51.405483 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 16:11:51.405908 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260924 16:11:51.407617 26846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:51.387470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51364 (local address 127.3.214.252:45929)
0924 16:11:51.387657 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:51.387664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:51.387738 (+ 74us) server_negotiation.cc:407] Connection header received
0924 16:11:51.387960 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:51.387964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:51.388057 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:51.388147 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:51.389349 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.390146 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.391801 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.392586 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.392716 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:51.397884 (+ 5168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:51.397894 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:51.397900 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:51.397971 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:51.402372 (+ 4401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:51.403676 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:51.403684 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:51.403688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:51.403901 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:51.404440 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:51.404446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:51.404449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:51.404886 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:51.405047 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:51.406864 (+ 1817us) server_negotiation.cc:299] Negotiation successful
0924 16:11:51.407198 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:11:51.415395 26694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041: Bootstrap starting.
I20260924 16:11:51.421183 26694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:51.423163 26694 log.cc:824] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:51.428501 26694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041: No bootstrap required, opened a new log
I20260924 16:11:51.439033 26694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8959891c254e6180c804046d250041" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34619 } } peers { permanent_uuid: "21f6f1d18fc44164a0434bee5677fcd3" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } } peers { permanent_uuid: "5d9cc96eaa4143eb90478e7ba18b2b4f" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } }
I20260924 16:11:51.439777 26694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c8959891c254e6180c804046d250041, State: Initialized, Role: FOLLOWER
W20260924 16:11:51.440168 26797 tablet.cc:2369] T 00000000000000000000000000000000 P 5d9cc96eaa4143eb90478e7ba18b2b4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:11:51.440551 26694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8959891c254e6180c804046d250041" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34619 } } peers { permanent_uuid: "21f6f1d18fc44164a0434bee5677fcd3" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } } peers { permanent_uuid: "5d9cc96eaa4143eb90478e7ba18b2b4f" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } }
I20260924 16:11:51.442415 26879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8959891c254e6180c804046d250041" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34619 } } peers { permanent_uuid: "21f6f1d18fc44164a0434bee5677fcd3" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } } peers { permanent_uuid: "5d9cc96eaa4143eb90478e7ba18b2b4f" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } } }
I20260924 16:11:51.443756 26879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:11:51.444046 26694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:51.444178 26669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 16:11:51.444458 26669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:51.448768 26669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8959891c254e6180c804046d250041" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34619 } } peers { permanent_uuid: "21f6f1d18fc44164a0434bee5677fcd3" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } } peers { permanent_uuid: "5d9cc96eaa4143eb90478e7ba18b2b4f" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } }
I20260924 16:11:51.453007 26743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c8959891c254e6180c804046d250041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "21f6f1d18fc44164a0434bee5677fcd3"
I20260924 16:11:51.453874 26743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21f6f1d18fc44164a0434bee5677fcd3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:51.459153 26743 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 21f6f1d18fc44164a0434bee5677fcd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c8959891c254e6180c804046d250041 in term 1.
I20260924 16:11:51.460530 26643 leader_election.cc:302] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f6f1d18fc44164a0434bee5677fcd3, 5c8959891c254e6180c804046d250041; no voters:
I20260924 16:11:51.461385 26879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:51.467500 26818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c8959891c254e6180c804046d250041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5d9cc96eaa4143eb90478e7ba18b2b4f"
I20260924 16:11:51.468290 26818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d9cc96eaa4143eb90478e7ba18b2b4f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:51.473203 26818 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5d9cc96eaa4143eb90478e7ba18b2b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c8959891c254e6180c804046d250041 in term 1.
I20260924 16:11:51.475044 26669 leader_election.cc:288] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041 [CANDIDATE]: Term 1 election: Requested vote from peers 21f6f1d18fc44164a0434bee5677fcd3 (127.3.214.253:36527), 5d9cc96eaa4143eb90478e7ba18b2b4f (127.3.214.252:45929)
I20260924 16:11:51.494867 26879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041 [term 1 LEADER]: Becoming Leader. State: Replica: 5c8959891c254e6180c804046d250041, State: Running, Role: LEADER
I20260924 16:11:51.495959 26879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8959891c254e6180c804046d250041" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34619 } } peers { permanent_uuid: "21f6f1d18fc44164a0434bee5677fcd3" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } } peers { permanent_uuid: "5d9cc96eaa4143eb90478e7ba18b2b4f" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } }
I20260924 16:11:51.514649 26881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c8959891c254e6180c804046d250041. Latest consensus state: current_term: 1 leader_uuid: "5c8959891c254e6180c804046d250041" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8959891c254e6180c804046d250041" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34619 } } peers { permanent_uuid: "21f6f1d18fc44164a0434bee5677fcd3" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } } peers { permanent_uuid: "5d9cc96eaa4143eb90478e7ba18b2b4f" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } } }
I20260924 16:11:51.515450 26881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:51.526042 26889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260924 16:11:51.531939 26722 tablet.cc:2369] T 00000000000000000000000000000000 P 21f6f1d18fc44164a0434bee5677fcd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:11:51.555073 26849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:51.508723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38472 (local address 127.3.214.253:36527)
0924 16:11:51.509055 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:51.509063 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:51.510894 (+ 1831us) server_negotiation.cc:407] Connection header received
0924 16:11:51.511675 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:51.511682 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:51.511778 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:51.511861 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:51.518488 (+ 6627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.519690 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.527723 (+ 8033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.528499 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.528623 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:51.541200 (+ 12577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:51.541210 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:51.541216 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:51.541283 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:51.545674 (+ 4391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:51.547233 (+ 1559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:51.547240 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:51.547243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:51.547436 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:51.550265 (+ 2829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:51.550273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:51.550277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:51.550808 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:51.551135 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:51.554379 (+ 3244us) server_negotiation.cc:299] Negotiation successful
0924 16:11:51.554691 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260924 16:11:51.559362 26889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:51.574719 26847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:51.510943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43696 (local address 127.3.214.254:34619)
0924 16:11:51.511170 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:51.511177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:51.527489 (+ 16312us) server_negotiation.cc:407] Connection header received
0924 16:11:51.528799 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:51.528809 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:51.528922 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:51.529015 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:51.535784 (+ 6769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.537029 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.547819 (+ 10790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.548571 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.548693 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:51.561497 (+ 12804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:51.561507 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:51.561512 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:51.561591 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:51.569930 (+ 8339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:51.571027 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:51.571034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:51.571039 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:51.571204 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:51.571820 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:51.571826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:51.571829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:51.572298 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:51.572616 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:51.573947 (+ 1331us) server_negotiation.cc:299] Negotiation successful
0924 16:11:51.574271 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 16:11:51.593962 26846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:51.536615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51376 (local address 127.3.214.252:45929)
0924 16:11:51.536881 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:51.536890 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:51.541107 (+ 4217us) server_negotiation.cc:407] Connection header received
0924 16:11:51.551912 (+ 10805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:51.551926 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:51.552185 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:51.552371 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:51.563125 (+ 10754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.564462 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.578744 (+ 14282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:51.579579 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:51.579691 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:51.584402 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:51.584411 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:51.584416 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:51.584471 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:51.588433 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:51.589448 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:51.589454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:51.589458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:51.589639 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:51.590142 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:51.590148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:51.590151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:51.590705 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:51.590981 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:51.593052 (+ 2071us) server_negotiation.cc:299] Negotiation successful
0924 16:11:51.593471 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 16:11:51.635286 26818 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d9cc96eaa4143eb90478e7ba18b2b4f [term 1 FOLLOWER]: Refusing update from remote peer 5c8959891c254e6180c804046d250041: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:51.637933 26743 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 21f6f1d18fc44164a0434bee5677fcd3 [term 1 FOLLOWER]: Refusing update from remote peer 5c8959891c254e6180c804046d250041: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:51.640422 26879 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d9cc96eaa4143eb90478e7ba18b2b4f" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:11:51.642795 26879 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f6f1d18fc44164a0434bee5677fcd3" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 16:11:51.676193 26648 tablet.cc:2369] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:11:51.693887 26851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9cc96eaa4143eb90478e7ba18b2b4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c8959891c254e6180c804046d250041. Latest consensus state: current_term: 1 leader_uuid: "5c8959891c254e6180c804046d250041" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8959891c254e6180c804046d250041" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34619 } } peers { permanent_uuid: "21f6f1d18fc44164a0434bee5677fcd3" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } } peers { permanent_uuid: "5d9cc96eaa4143eb90478e7ba18b2b4f" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } } }
I20260924 16:11:51.694386 26851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9cc96eaa4143eb90478e7ba18b2b4f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:11:51.699628 26865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21f6f1d18fc44164a0434bee5677fcd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c8959891c254e6180c804046d250041. Latest consensus state: current_term: 1 leader_uuid: "5c8959891c254e6180c804046d250041" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8959891c254e6180c804046d250041" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34619 } } peers { permanent_uuid: "21f6f1d18fc44164a0434bee5677fcd3" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } } peers { permanent_uuid: "5d9cc96eaa4143eb90478e7ba18b2b4f" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } } }
I20260924 16:11:51.700083 26865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21f6f1d18fc44164a0434bee5677fcd3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:11:51.707684 26851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9cc96eaa4143eb90478e7ba18b2b4f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c8959891c254e6180c804046d250041" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8959891c254e6180c804046d250041" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34619 } } peers { permanent_uuid: "21f6f1d18fc44164a0434bee5677fcd3" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } } peers { permanent_uuid: "5d9cc96eaa4143eb90478e7ba18b2b4f" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } } }
I20260924 16:11:51.708060 26851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9cc96eaa4143eb90478e7ba18b2b4f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:11:51.709609 26881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c8959891c254e6180c804046d250041" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8959891c254e6180c804046d250041" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34619 } } peers { permanent_uuid: "21f6f1d18fc44164a0434bee5677fcd3" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } } peers { permanent_uuid: "5d9cc96eaa4143eb90478e7ba18b2b4f" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } } }
I20260924 16:11:51.710011 26881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:51.713277 26906 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 16:11:51.724787 26879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c8959891c254e6180c804046d250041" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8959891c254e6180c804046d250041" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34619 } } peers { permanent_uuid: "21f6f1d18fc44164a0434bee5677fcd3" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } } peers { permanent_uuid: "5d9cc96eaa4143eb90478e7ba18b2b4f" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } } }
I20260924 16:11:51.725235 26879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:51.729116 26889 catalog_manager.cc:1347] Generated new cluster ID: 9947fc4146fb419f837d76ac9b5feeaa
I20260924 16:11:51.729310 26889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:51.740356 26900 mvcc.cc:205] Tried to move back new op lower bound from 7332930812435333120 to 7332930811916218368. Current Snapshot: MvccSnapshot[applied={T|T < 7332930812435333120}]
I20260924 16:11:51.753147 26865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21f6f1d18fc44164a0434bee5677fcd3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c8959891c254e6180c804046d250041" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8959891c254e6180c804046d250041" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 34619 } } peers { permanent_uuid: "21f6f1d18fc44164a0434bee5677fcd3" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 36527 } } peers { permanent_uuid: "5d9cc96eaa4143eb90478e7ba18b2b4f" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 45929 } } }
I20260924 16:11:51.753551 26865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21f6f1d18fc44164a0434bee5677fcd3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:11:51.789790 26889 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:51.792490 26889 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:51.822495 26889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c8959891c254e6180c804046d250041: Generated new TSK 0
I20260924 16:11:51.824209 26889 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:11:52.003806 26849 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:51.988796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38480 (local address 127.3.214.253:36527)
0924 16:11:51.989039 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:51.989047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:52.001469 (+ 12422us) server_negotiation.cc:407] Connection header received
0924 16:11:52.001901 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:52.001908 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:52.002017 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:52.002118 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:52.003517 (+ 1399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38480: BlockingRecv error: recv got EOF from 127.0.0.1:38480 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 16:11:52.007189 26847 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:51.997150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43708 (local address 127.3.214.254:34619)
0924 16:11:51.997528 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:51.997536 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:52.004784 (+ 7248us) server_negotiation.cc:407] Connection header received
0924 16:11:52.005112 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:52.005119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:52.005215 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:52.005309 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:52.006902 (+ 1593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43708: BlockingRecv error: recv got EOF from 127.0.0.1:43708 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
W20260924 16:11:52.008008 26846 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:52.005447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51386 (local address 127.3.214.252:45929)
0924 16:11:52.005703 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:52.005709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:52.006036 (+ 327us) server_negotiation.cc:407] Connection header received
0924 16:11:52.006408 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:52.006415 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:52.006514 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:52.006588 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:52.007691 (+ 1103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51386: BlockingRecv error: recv got EOF from 127.0.0.1:51386 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260924 16:11:52.018227 26849 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:52.014935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38488 (local address 127.3.214.253:36527)
0924 16:11:52.015167 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:52.015173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:52.015828 (+ 655us) server_negotiation.cc:407] Connection header received
0924 16:11:52.016265 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:52.016273 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:52.016445 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:52.016580 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:52.017844 (+ 1264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38488: BlockingRecv error: recv got EOF from 127.0.0.1:38488 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260924 16:11:52.033353 26849 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:52.029935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38504 (local address 127.3.214.253:36527)
0924 16:11:52.030225 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:52.030232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:52.030986 (+ 754us) server_negotiation.cc:407] Connection header received
0924 16:11:52.031284 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:52.031290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:52.031421 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:52.031555 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:52.033027 (+ 1472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38504: BlockingRecv error: recv got EOF from 127.0.0.1:38504 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260924 16:11:52.043807 26849 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:52.041013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38520 (local address 127.3.214.253:36527)
0924 16:11:52.041276 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:52.041283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:52.041954 (+ 671us) server_negotiation.cc:407] Connection header received
0924 16:11:52.042257 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:52.042262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:52.042360 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:52.042475 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:52.043468 (+ 993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38520: BlockingRecv error: recv got EOF from 127.0.0.1:38520 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260924 16:11:52.055357 26849 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:52.052732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38526 (local address 127.3.214.253:36527)
0924 16:11:52.053011 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:52.053017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:52.053521 (+ 504us) server_negotiation.cc:407] Connection header received
0924 16:11:52.053808 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:52.053813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:52.053907 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:52.054025 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:52.055073 (+ 1048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38526: BlockingRecv error: recv got EOF from 127.0.0.1:38526 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
W20260924 16:11:52.071512 26849 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:52.068421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38534 (local address 127.3.214.253:36527)
0924 16:11:52.068718 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:52.068725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:52.069403 (+ 678us) server_negotiation.cc:407] Connection header received
0924 16:11:52.069591 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:52.069597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:52.069745 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:52.069920 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:52.071224 (+ 1304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38534: BlockingRecv error: recv got EOF from 127.0.0.1:38534 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
W20260924 16:11:52.095806 26849 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:52.092783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38544 (local address 127.3.214.253:36527)
0924 16:11:52.093060 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:52.093067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:52.093543 (+ 476us) server_negotiation.cc:407] Connection header received
0924 16:11:52.093844 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:52.093849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:52.093958 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:52.094089 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:52.095472 (+ 1383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38544: BlockingRecv error: recv got EOF from 127.0.0.1:38544 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
W20260924 16:11:52.136284 26849 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:52.133560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38550 (local address 127.3.214.253:36527)
0924 16:11:52.133855 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:52.133863 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:52.134322 (+ 459us) server_negotiation.cc:407] Connection header received
0924 16:11:52.134557 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:52.134564 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:52.134719 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:52.134881 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:52.135922 (+ 1041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38550: BlockingRecv error: recv got EOF from 127.0.0.1:38550 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260924 16:11:52.208068 26849 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:52.205226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38558 (local address 127.3.214.253:36527)
0924 16:11:52.205591 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:52.205599 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:52.206051 (+ 452us) server_negotiation.cc:407] Connection header received
0924 16:11:52.206381 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:52.206386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:52.206466 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:52.206569 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:52.207750 (+ 1181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38558: BlockingRecv error: recv got EOF from 127.0.0.1:38558 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
W20260924 16:11:52.343261 26849 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:52.340733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38562 (local address 127.3.214.253:36527)
0924 16:11:52.341103 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:52.341112 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:52.341523 (+ 411us) server_negotiation.cc:407] Connection header received
0924 16:11:52.341729 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:52.341735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:52.341841 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:52.341978 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:52.342920 (+ 942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38562: BlockingRecv error: recv got EOF from 127.0.0.1:38562 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
I20260924 16:11:52.345098 26863 catalog_manager.cc:1259] Loaded cluster ID: 9947fc4146fb419f837d76ac9b5feeaa
I20260924 16:11:52.345239 26863 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5d9cc96eaa4143eb90478e7ba18b2b4f: loading cluster ID for follower catalog manager: success
I20260924 16:11:52.348742 26863 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5d9cc96eaa4143eb90478e7ba18b2b4f: acquiring CA information for follower catalog manager: success
I20260924 16:11:52.351223 26863 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5d9cc96eaa4143eb90478e7ba18b2b4f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:11:52.367597 26876 catalog_manager.cc:1259] Loaded cluster ID: 9947fc4146fb419f837d76ac9b5feeaa
I20260924 16:11:52.367730 26876 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 21f6f1d18fc44164a0434bee5677fcd3: loading cluster ID for follower catalog manager: success
I20260924 16:11:52.370875 26876 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 21f6f1d18fc44164a0434bee5677fcd3: acquiring CA information for follower catalog manager: success
I20260924 16:11:52.373276 26876 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 21f6f1d18fc44164a0434bee5677fcd3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:11:52.611057 26849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:52.603857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38566 (local address 127.3.214.253:36527)
0924 16:11:52.604137 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:52.604143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:52.604233 (+ 90us) server_negotiation.cc:407] Connection header received
0924 16:11:52.604510 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:52.604517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:52.604756 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:52.604917 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:52.605469 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:52.606387 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:52.607950 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:52.608447 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:52.608562 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:52.609418 (+ 856us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:52.610284 (+ 866us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:52.610452 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:52.610651 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 16:11:52.610750 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 16:11:52.625455 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:52.618846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51394 (local address 127.3.214.252:45929)
0924 16:11:52.619446 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:52.619453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:52.619477 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:11:52.619809 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:52.619814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:52.620032 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:52.620219 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:52.620654 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:52.621447 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:52.622922 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:52.623348 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:52.623462 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:52.624028 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:52.624739 (+ 711us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:52.624898 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:52.624985 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 16:11:52.625084 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":316,"threads_started":1}
I20260924 16:11:52.642817 26980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:52.633957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43724 (local address 127.3.214.254:34619)
0924 16:11:52.634601 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:52.634609 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:52.635474 (+ 865us) server_negotiation.cc:407] Connection header received
0924 16:11:52.635785 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:52.635795 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:52.636027 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:52.636217 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:52.637017 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:52.637813 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:52.640086 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:52.640708 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:52.640831 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:52.641376 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:52.641904 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:52.642017 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:52.642293 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 16:11:52.642429 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":314,"threads_started":1}
I20260924 16:11:52.642848 26849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:52.634142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38568 (local address 127.3.214.253:36527)
0924 16:11:52.634426 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:52.634435 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:52.635475 (+ 1040us) server_negotiation.cc:407] Connection header received
0924 16:11:52.635937 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:52.635946 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:52.636076 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:52.636201 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:52.636966 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:52.638340 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:52.640626 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:52.641340 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:52.641445 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:52.641931 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:52.642061 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:52.642191 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:52.642443 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 16:11:52.642538 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260924 16:11:52.646435 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:52.635473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51404 (local address 127.3.214.252:45929)
0924 16:11:52.635693 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:52.635700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:52.638914 (+ 3214us) server_negotiation.cc:407] Connection header received
0924 16:11:52.639102 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:52.639108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:52.639215 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:52.639330 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:52.640120 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:52.641050 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:52.643990 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:52.644646 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:52.644798 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:52.645428 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:52.645611 (+ 183us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:52.645734 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:52.645889 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 16:11:52.646060 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 16:11:52.655014 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 26627
I20260924 16:11:52.656908 26689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:52.786536 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 26627
I20260924 16:11:52.807271 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 26701
I20260924 16:11:52.808876 26763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:52.925050 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 26701
I20260924 16:11:52.944952 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 26776
I20260924 16:11:52.946422 26838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:53.067360 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 26776
2026-09-24T16:11:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5110 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:53 dist-test-slave-1gc3 krb5kdc[26989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:53 dist-test-slave-1gc3 krb5kdc[26989](info): set up 2 sockets
Sep 24 16:11:53 dist-test-slave-1gc3 krb5kdc[26989](info): commencing operation
krb5kdc: starting...
W20260924 16:11:54.152282 3931 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/test-user.keytab.
Sep 24 16:11:54 dist-test-slave-1gc3 krb5kdc[26989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.214.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
Entry for principal kudu/127.3.214.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab.
I20260924 16:11:54.490482 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:42565
--webserver_interface=127.3.214.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:43577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:42565,127.3.214.253:39981,127.3.214.252:44423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:54.919559 27002 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:11:54.920019 27002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:54.920143 27002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:54.920274 27002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:54.928617 27002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:54.928766 27002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:54.928831 27002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:54.928910 27002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:54.929008 27002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:54.941010 27002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:43577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:42565,127.3.214.253:39981,127.3.214.252:44423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.254.keytab
--principal=kudu/127.3.214.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.254:42565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.214.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:54.942970 27002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:54.944832 27002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:54.960474 27008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:54.960445 27010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:54.960718 27007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:54.961640 27002 server_base.cc:1034] running on GCE node
I20260924 16:11:54.962890 27002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:54.964932 27002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:54.966286 27002 hybrid_clock.cc:648] HybridClock initialized: now 1790266314966178 us; error 54 us; skew 500 ppm
I20260924 16:11:54.966758 27002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:54 dist-test-slave-1gc3 krb5kdc[26989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266314, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:54.970854 27002 init.cc:377] Logged in from keytab as kudu/127.3.214.254@KRBTEST.COM (short username kudu)
I20260924 16:11:54.973026 27002 webserver.cc:460] Webserver started at http://127.3.214.254:37693/ using document root <none> and password file <none>
I20260924 16:11:54.974045 27002 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:54.974247 27002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:54.974673 27002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:54.978865 27002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-0/data/instance:
uuid: "64bf3d337d5548bd8bb165e102c93a63"
format_stamp: "Formatted at 2026-09-24 16:11:54 on dist-test-slave-1gc3"
server_key: "84b52c4056ec031e0d4daa7bd89c9e09"
server_key_iv: "b810e356441609c7ac69a0877b71fdfd"
server_key_version: "encryptionkey@0"
I20260924 16:11:54.980036 27002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-0/wal/instance:
uuid: "64bf3d337d5548bd8bb165e102c93a63"
format_stamp: "Formatted at 2026-09-24 16:11:54 on dist-test-slave-1gc3"
server_key: "84b52c4056ec031e0d4daa7bd89c9e09"
server_key_iv: "b810e356441609c7ac69a0877b71fdfd"
server_key_version: "encryptionkey@0"
I20260924 16:11:54.987460 27002 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260924 16:11:54.992468 27017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:54.994967 27002 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260924 16:11:54.995306 27002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-0/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-0/wal
uuid: "64bf3d337d5548bd8bb165e102c93a63"
format_stamp: "Formatted at 2026-09-24 16:11:54 on dist-test-slave-1gc3"
server_key: "84b52c4056ec031e0d4daa7bd89c9e09"
server_key_iv: "b810e356441609c7ac69a0877b71fdfd"
server_key_version: "encryptionkey@0"
I20260924 16:11:54.995654 27002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:55.017184 27002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:55.023110 27002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:55.023530 27002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:55.047092 27002 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.254:42565
I20260924 16:11:55.047122 27068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.254:42565 every 8 connection(s)
I20260924 16:11:55.048868 27002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:55.053757 27069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:55.056638 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 27002
I20260924 16:11:55.056843 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-0/wal/instance
I20260924 16:11:55.057425 3931 external_mini_cluster.cc:1428] Setting key ae9f066a7cc6293427678051f2b6b423
I20260924 16:11:55.063658 27069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42565 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:11:55 dist-test-slave-1gc3 krb5kdc[26989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266314, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.214.253@KRBTEST.COM; defaulting to no policy
I20260924 16:11:55.098511 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.068859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:42565 (local address 127.0.0.1:49676)
0924 16:11:55.070388 (+ 1529us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.070467 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:55.072703 (+ 2236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:55.074392 (+ 1689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:55.074400 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:55.075535 (+ 1135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:55.076043 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:55.076073 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:55.078709 (+ 2636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:55.078715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:55.080157 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:55.080185 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:55.081889 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:55.081897 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:55.082135 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:55.083030 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:55.083063 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:55.086177 (+ 3114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:55.091885 (+ 5708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:55.091909 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:55.091924 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:55.092612 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:55.093481 (+ 869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:55.093486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:55.093489 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:55.093752 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:55.095546 (+ 1794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:55.095557 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:55.095976 (+ 419us) client_negotiation.cc:769] Sending connection context
0924 16:11:55.096343 (+ 367us) client_negotiation.cc:240] Negotiation successful
0924 16:11:55.096966 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"thread_start_us":611,"threads_started":1}
I20260924 16:11:55.098706 27072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.068859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49676 (local address 127.3.214.254:42565)
0924 16:11:55.070243 (+ 1384us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:55.070262 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:55.070636 (+ 374us) server_negotiation.cc:407] Connection header received
0924 16:11:55.073351 (+ 2715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:55.073364 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:55.073900 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:55.074260 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:55.076403 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:55.078373 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:55.080414 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:55.081440 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:55.081697 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:55.086626 (+ 4929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:55.086642 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:55.086672 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:55.086739 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:55.091533 (+ 4794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:55.092872 (+ 1339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:55.092887 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:55.092891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:55.093232 (+ 341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:55.093896 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:55.093902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:55.093905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:55.095145 (+ 1240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:55.095501 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:55.096628 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0924 16:11:55.097196 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":172,"server-negotiator.queue_time_us":1045,"spinlock_wait_cycles":79616,"thread_start_us":1247,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.253@KRBTEST.COM" created.
I20260924 16:11:55.103062 27069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:11:55.104714 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.103974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:39981 (local address 127.0.0.1:56604)
0924 16:11:55.104303 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.104526 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:39981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260924 16:11:55.105268 27021 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.214.253:39981: connect: Connection refused (error 111)
I20260924 16:11:55.107807 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.107005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:39981 (local address 127.0.0.1:56620)
0924 16:11:55.107403 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.107573 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:39981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
W20260924 16:11:55.108181 27069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.253:39981: Network error: Client connection negotiation failed: client connection to 127.3.214.253:39981: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.253.keytab.
Entry for principal kudu/127.3.214.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.253.keytab.
I20260924 16:11:55.141999 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.253:39981
--webserver_interface=127.3.214.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.253.keytab
--principal=kudu/127.3.214.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:43577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:42565,127.3.214.253:39981,127.3.214.252:44423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:11:55.157688 27069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } attempt: 1
I20260924 16:11:55.159397 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.158742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:39981 (local address 127.0.0.1:56632)
0924 16:11:55.159063 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.159238 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:39981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260924 16:11:55.160946 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.160482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:39981 (local address 127.0.0.1:56636)
0924 16:11:55.160679 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.160769 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:39981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260924 16:11:55.161164 27069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.253:39981: Network error: Client connection negotiation failed: client connection to 127.3.214.253:39981: connect: Connection refused (error 111)
I20260924 16:11:55.229540 27069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } attempt: 2
I20260924 16:11:55.231359 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.230637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:39981 (local address 127.0.0.1:56638)
0924 16:11:55.231038 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.231185 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:39981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260924 16:11:55.232847 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.232339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:39981 (local address 127.0.0.1:56650)
0924 16:11:55.232573 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.232677 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:39981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260924 16:11:55.233088 27069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.253:39981: Network error: Client connection negotiation failed: client connection to 127.3.214.253:39981: connect: Connection refused (error 111)
I20260924 16:11:55.324407 27069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } attempt: 3
I20260924 16:11:55.325963 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.325177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:39981 (local address 127.0.0.1:56662)
0924 16:11:55.325526 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.325697 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:39981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260924 16:11:55.327387 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.326908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:39981 (local address 127.0.0.1:56678)
0924 16:11:55.327078 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.327186 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:39981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 16:11:55.327636 27069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.253:39981: Network error: Client connection negotiation failed: client connection to 127.3.214.253:39981: connect: Connection refused (error 111)
I20260924 16:11:55.471007 27069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } attempt: 4
I20260924 16:11:55.472846 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.471976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:39981 (local address 127.0.0.1:56688)
0924 16:11:55.472416 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.472601 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:39981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229}
I20260924 16:11:55.474386 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.473823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:39981 (local address 127.0.0.1:56690)
0924 16:11:55.474025 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.474192 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.253:39981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260924 16:11:55.474692 27069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.253:39981: Network error: Client connection negotiation failed: client connection to 127.3.214.253:39981: connect: Connection refused (error 111)
W20260924 16:11:55.568452 27076 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:11:55.568936 27076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:55.569010 27076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:55.569108 27076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:55.577574 27076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:55.577725 27076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:55.577788 27076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:55.577848 27076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:55.577922 27076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:55.589712 27076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:43577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:42565,127.3.214.253:39981,127.3.214.252:44423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.253.keytab
--principal=kudu/127.3.214.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.253:39981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.214.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:55.591670 27076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:55.593353 27076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:55.608211 27084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:55.608042 27082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:55.608512 27081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:55.609596 27076 server_base.cc:1034] running on GCE node
I20260924 16:11:55.610685 27076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:55.612572 27076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:55.613909 27076 hybrid_clock.cc:648] HybridClock initialized: now 1790266315613797 us; error 59 us; skew 500 ppm
I20260924 16:11:55.614600 27076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:55 dist-test-slave-1gc3 krb5kdc[26989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:55.618765 27076 init.cc:377] Logged in from keytab as kudu/127.3.214.253@KRBTEST.COM (short username kudu)
I20260924 16:11:55.620779 27076 webserver.cc:460] Webserver started at http://127.3.214.253:39183/ using document root <none> and password file <none>
I20260924 16:11:55.621672 27076 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:55.621794 27076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:55.622211 27076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:55.625748 27076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-1/data/instance:
uuid: "9f3335f30abb40739b6e58da9db3563f"
format_stamp: "Formatted at 2026-09-24 16:11:55 on dist-test-slave-1gc3"
server_key: "2aa08dbe0abd99a753b23b194a9bf66d"
server_key_iv: "e81b90cbf0796d538e9b5b5a4812ff0b"
server_key_version: "encryptionkey@0"
I20260924 16:11:55.626765 27076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-1/wal/instance:
uuid: "9f3335f30abb40739b6e58da9db3563f"
format_stamp: "Formatted at 2026-09-24 16:11:55 on dist-test-slave-1gc3"
server_key: "2aa08dbe0abd99a753b23b194a9bf66d"
server_key_iv: "e81b90cbf0796d538e9b5b5a4812ff0b"
server_key_version: "encryptionkey@0"
I20260924 16:11:55.633455 27076 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 16:11:55.637971 27091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:55.640228 27076 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:11:55.640507 27076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-1/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-1/wal
uuid: "9f3335f30abb40739b6e58da9db3563f"
format_stamp: "Formatted at 2026-09-24 16:11:55 on dist-test-slave-1gc3"
server_key: "2aa08dbe0abd99a753b23b194a9bf66d"
server_key_iv: "e81b90cbf0796d538e9b5b5a4812ff0b"
server_key_version: "encryptionkey@0"
I20260924 16:11:55.640784 27076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:55.662928 27076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:55.668880 27076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:55.669203 27076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:55.691907 27076 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.253:39981
I20260924 16:11:55.691979 27142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.253:39981 every 8 connection(s)
I20260924 16:11:55.693523 27076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-1/data/info.pb
I20260924 16:11:55.698971 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:55.699532 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 27076
I20260924 16:11:55.699760 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-1/wal/instance
I20260924 16:11:55.700398 3931 external_mini_cluster.cc:1428] Setting key 008aa7942097b38d7998113360b1dc47
I20260924 16:11:55.710666 27143 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42565 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:11:55 dist-test-slave-1gc3 krb5kdc[26989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.253@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.214.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.214.252@KRBTEST.COM" created.
I20260924 16:11:55.740249 27145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.715436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49678 (local address 127.3.214.254:42565)
0924 16:11:55.716270 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:55.716277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:55.717610 (+ 1333us) server_negotiation.cc:407] Connection header received
0924 16:11:55.720316 (+ 2706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:55.720322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:55.720440 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:55.720585 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:55.723201 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:55.724197 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:55.726746 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:55.727280 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:55.727423 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:55.732483 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:55.732490 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:55.732494 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:55.732551 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:55.735679 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:55.736859 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:55.736864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:55.736866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:55.737009 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:55.737620 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:55.737628 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:55.737630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:55.738052 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:55.738337 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:55.739345 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0924 16:11:55.739699 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":380,"threads_started":1}
I20260924 16:11:55.741329 27146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.716336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:42565 (local address 127.0.0.1:49678)
0924 16:11:55.717332 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.717392 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:55.719845 (+ 2453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:55.720940 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:55.720954 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:55.721824 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:55.722768 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:55.722798 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:55.724470 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:55.724475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:55.726488 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:55.726516 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:55.727613 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:55.727621 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:55.727941 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:55.728842 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:55.728876 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:55.732087 (+ 3211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:55.735932 (+ 3845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:55.735945 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:55.735969 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:55.736706 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:55.737123 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:55.737127 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:55.737129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:55.737361 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:55.738345 (+ 984us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:55.738358 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:55.738831 (+ 473us) client_negotiation.cc:769] Sending connection context
0924 16:11:55.739155 (+ 324us) client_negotiation.cc:240] Negotiation successful
0924 16:11:55.739798 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":387,"threads_started":1}
I20260924 16:11:55.743887 27143 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:11:55 dist-test-slave-1gc3 krb5kdc[26989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.253@KRBTEST.COM for kudu/127.3.214.253@KRBTEST.COM
I20260924 16:11:55.760244 27146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.745080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:39981 (local address 127.0.0.1:56694)
0924 16:11:55.745364 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.745384 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:55.745514 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:55.746961 (+ 1447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:55.746965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:55.747485 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:55.747628 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:55.747638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:55.749112 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:55.749116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:55.749952 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:55.749965 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:55.750889 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:55.750895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:55.751093 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:55.751682 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:55.751700 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:55.753928 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:55.757481 (+ 3553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:55.757487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:55.757491 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:55.757883 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:55.758357 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:55.758363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:55.758365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:55.758487 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:55.759130 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:55.759135 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:55.759288 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 16:11:55.759409 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 16:11:55.759733 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 16:11:55.760351 27148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.745369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56694 (local address 127.3.214.253:39981)
0924 16:11:55.746419 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:55.746429 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:55.746470 (+ 41us) server_negotiation.cc:407] Connection header received
0924 16:11:55.746548 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:55.746555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:55.746789 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:55.746953 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:55.747817 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:55.748958 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:55.750133 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:55.750741 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:55.750856 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:55.754221 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:55.754240 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:55.754260 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:55.754294 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:55.757300 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:55.758026 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:55.758035 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:55.758049 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:55.758225 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:55.758593 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:55.758598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:55.758601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:55.758985 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:55.759199 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:55.759583 (+ 384us) server_negotiation.cc:299] Negotiation successful
0924 16:11:55.759862 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":783,"thread_start_us":544,"threads_started":1}
I20260924 16:11:55.763087 27143 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:11:55.764616 27146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.764079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:44423 (local address 127.0.0.1:38438)
0924 16:11:55.764300 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.764424 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260924 16:11:55.765043 27094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
I20260924 16:11:55.767256 27146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.766819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:44423 (local address 127.0.0.1:38448)
0924 16:11:55.767003 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.767117 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260924 16:11:55.767554 27143 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:44423: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
I20260924 16:11:55.773957 27069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } attempt: 5
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.214.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.252.keytab.
Entry for principal kudu/127.3.214.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.252.keytab.
I20260924 16:11:55.781883 3931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
/tmp/dist-test-taskHlT5a5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.252:44423
--webserver_interface=127.3.214.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.252.keytab
--principal=kudu/127.3.214.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.214.212:43577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.214.254:42565,127.3.214.253:39981,127.3.214.252:44423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:11:55 dist-test-slave-1gc3 krb5kdc[26989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266314, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for kudu/127.3.214.253@KRBTEST.COM
I20260924 16:11:55.789680 27148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.775302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.3.214.253:39981)
0924 16:11:55.775584 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:55.775592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:55.775622 (+ 30us) server_negotiation.cc:407] Connection header received
0924 16:11:55.775802 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:55.775809 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:55.775907 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:55.776068 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:55.777170 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:55.778065 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:55.779514 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:55.780042 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:55.780205 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:55.783820 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:55.783827 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:55.783830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:55.783878 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:55.786781 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:55.787557 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:55.787563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:55.787565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:55.787709 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:55.788102 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:55.788106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:55.788108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:55.788466 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:55.788614 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:55.789034 (+ 420us) server_negotiation.cc:299] Negotiation successful
0924 16:11:55.789312 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260924 16:11:55.789680 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.775146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:39981 (local address 127.0.0.1:56704)
0924 16:11:55.775440 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.775472 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:55.775630 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:55.776179 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:55.776183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:55.776690 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:55.776958 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:55.776968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:55.778278 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:55.778283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:55.779318 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:55.779331 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:55.780240 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:55.780246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:55.780440 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:55.781157 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:55.781177 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:55.783538 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:55.786979 (+ 3441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:55.786985 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:55.786990 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:55.787418 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:55.787828 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:55.787834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:55.787838 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:55.787994 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:55.788585 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:55.788591 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:55.788771 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 16:11:55.788901 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 16:11:55.789298 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260924 16:11:55.791319 27069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:11:55.793049 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.792416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:44423 (local address 127.0.0.1:38464)
0924 16:11:55.792694 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.792816 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260924 16:11:55.794411 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.793980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:44423 (local address 127.0.0.1:38478)
0924 16:11:55.794190 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.794271 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 16:11:55.794653 27069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:44423: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
I20260924 16:11:55.816958 27143 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } attempt: 1
I20260924 16:11:55.818172 27146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.817660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:44423 (local address 127.0.0.1:38486)
0924 16:11:55.817864 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.817972 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 16:11:55.819132 27146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.818790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:44423 (local address 127.0.0.1:38500)
0924 16:11:55.818936 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.819024 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 16:11:55.819331 27143 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:44423: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
I20260924 16:11:55.845914 27069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } attempt: 1
I20260924 16:11:55.847196 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.846710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:44423 (local address 127.0.0.1:38510)
0924 16:11:55.846916 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.847024 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 16:11:55.848184 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.847845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:44423 (local address 127.0.0.1:38518)
0924 16:11:55.848000 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.848072 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 16:11:55.848407 27069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:44423: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
I20260924 16:11:55.887609 27143 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } attempt: 2
I20260924 16:11:55.889268 27146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.888643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:44423 (local address 127.0.0.1:38520)
0924 16:11:55.888964 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.889089 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260924 16:11:55.890506 27146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.890071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:44423 (local address 127.0.0.1:38524)
0924 16:11:55.890281 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.890364 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260924 16:11:55.890735 27143 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:44423: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
I20260924 16:11:55.916673 27069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } attempt: 2
I20260924 16:11:55.918286 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.917602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:44423 (local address 127.0.0.1:38530)
0924 16:11:55.917934 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.918072 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260924 16:11:55.919420 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.919043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:44423 (local address 127.0.0.1:38534)
0924 16:11:55.919220 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.919294 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 16:11:55.919652 27069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:44423: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
I20260924 16:11:55.982012 27143 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } attempt: 3
I20260924 16:11:55.983345 27146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.982787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:44423 (local address 127.0.0.1:38540)
0924 16:11:55.983065 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.983167 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 16:11:55.984997 27146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.984392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:44423 (local address 127.0.0.1:38552)
0924 16:11:55.984735 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:55.984845 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225}
W20260924 16:11:55.985302 27143 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:44423: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
I20260924 16:11:56.025941 27069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } attempt: 3
I20260924 16:11:56.027269 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:56.026748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:44423 (local address 127.0.0.1:38560)
0924 16:11:56.026980 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:56.027085 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 16:11:56.029073 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:56.028463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:44423 (local address 127.0.0.1:38562)
0924 16:11:56.028791 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:56.028935 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
W20260924 16:11:56.029402 27069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:44423: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
I20260924 16:11:56.128574 27143 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } attempt: 4
I20260924 16:11:56.130297 27146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:56.129528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:44423 (local address 127.0.0.1:38566)
0924 16:11:56.129924 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:56.130054 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260924 16:11:56.131417 27146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:56.131042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:44423 (local address 127.0.0.1:38574)
0924 16:11:56.131200 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:56.131283 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 16:11:56.131644 27143 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:44423: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
I20260924 16:11:56.206665 27069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } attempt: 4
I20260924 16:11:56.208158 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:56.207530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:44423 (local address 127.0.0.1:38588)
0924 16:11:56.207780 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:56.207920 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 16:11:56.209795 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:56.209170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:44423 (local address 127.0.0.1:38594)
0924 16:11:56.209420 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:56.209562 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260924 16:11:56.210217 27069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.214.252:44423: Network error: Client connection negotiation failed: client connection to 127.3.214.252:44423: connect: Connection refused (error 111)
W20260924 16:11:56.217548 27151 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:11:56.218025 27151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:56.218154 27151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:56.218286 27151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:56.227164 27151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:56.227319 27151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:56.227391 27151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:56.227465 27151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:56.227571 27151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:56.240330 27151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.214.212:43577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.214.254:42565,127.3.214.253:39981,127.3.214.252:44423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/krb5kdc/kudu_127.3.214.252.keytab
--principal=kudu/127.3.214.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.214.252:44423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.214.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:11:56.242244 27151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:56.244195 27151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:56.260239 27156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:56.260124 27157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:56.260178 27159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:56.261530 27151 server_base.cc:1034] running on GCE node
I20260924 16:11:56.262717 27151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:56.264726 27151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:56.266054 27151 hybrid_clock.cc:648] HybridClock initialized: now 1790266316265936 us; error 72 us; skew 500 ppm
I20260924 16:11:56.266762 27151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:11:56 dist-test-slave-1gc3 krb5kdc[26989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266316, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:56.271181 27151 init.cc:377] Logged in from keytab as kudu/127.3.214.252@KRBTEST.COM (short username kudu)
I20260924 16:11:56.273373 27151 webserver.cc:460] Webserver started at http://127.3.214.252:39513/ using document root <none> and password file <none>
I20260924 16:11:56.274389 27151 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:56.274516 27151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:56.274892 27151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:56.278817 27151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-2/data/instance:
uuid: "481f86d1f74c44b2b5cafcf7347e5741"
format_stamp: "Formatted at 2026-09-24 16:11:56 on dist-test-slave-1gc3"
server_key: "7b52db8fefe7b30a15cac3e1e3d0ce26"
server_key_iv: "ac7d844d5024d03a4faf58ab837068f8"
server_key_version: "encryptionkey@0"
I20260924 16:11:56.279728 27151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-2/wal/instance:
uuid: "481f86d1f74c44b2b5cafcf7347e5741"
format_stamp: "Formatted at 2026-09-24 16:11:56 on dist-test-slave-1gc3"
server_key: "7b52db8fefe7b30a15cac3e1e3d0ce26"
server_key_iv: "ac7d844d5024d03a4faf58ab837068f8"
server_key_version: "encryptionkey@0"
I20260924 16:11:56.286346 27151 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 16:11:56.291091 27166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:56.293078 27151 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:11:56.293330 27151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-2/data,/tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-2/wal
uuid: "481f86d1f74c44b2b5cafcf7347e5741"
format_stamp: "Formatted at 2026-09-24 16:11:56 on dist-test-slave-1gc3"
server_key: "7b52db8fefe7b30a15cac3e1e3d0ce26"
server_key_iv: "ac7d844d5024d03a4faf58ab837068f8"
server_key_version: "encryptionkey@0"
I20260924 16:11:56.293604 27151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:56.316568 27151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:56.322234 27151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:56.322566 27151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:56.347827 27151 rpc_server.cc:307] RPC server started. Bound to: 127.3.214.252:44423
I20260924 16:11:56.347898 27217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.214.252:44423 every 8 connection(s)
I20260924 16:11:56.349704 27151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-2/data/info.pb
I20260924 16:11:56.350605 3931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu as pid 27151
I20260924 16:11:56.350808 3931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHlT5a5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790265920616583-3931-0/minicluster-data/master-2/wal/instance
I20260924 16:11:56.351438 3931 external_mini_cluster.cc:1428] Setting key 5178f1a5c5cd99203fe0e9cbc9fae40c
I20260924 16:11:56.355537 27218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:56.365978 27218 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42565 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:11:56 dist-test-slave-1gc3 krb5kdc[26989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.252@KRBTEST.COM
I20260924 16:11:56.380609 27221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:56.353571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38602 (local address 127.3.214.252:44423)
0924 16:11:56.354638 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:56.354651 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:56.354705 (+ 54us) server_negotiation.cc:407] Connection header received
0924 16:11:56.356647 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:56.356680 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:56.357191 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:56.357599 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:56.359788 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.361387 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.363583 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.364452 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.364653 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:56.370347 (+ 5694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:56.370360 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:56.370384 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:56.370476 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:56.374845 (+ 4369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:56.376019 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:56.376029 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:56.376040 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:56.376301 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:56.376788 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:56.376792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:56.376794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:56.377683 (+ 889us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:56.378125 (+ 442us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:56.378542 (+ 417us) server_negotiation.cc:299] Negotiation successful
0924 16:11:56.379146 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":534,"threads_started":1}
Sep 24 16:11:56 dist-test-slave-1gc3 krb5kdc[26989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266316, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.252@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
I20260924 16:11:56.389556 27222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:56.370190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49692 (local address 127.3.214.254:42565)
0924 16:11:56.370859 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:56.370866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:56.371754 (+ 888us) server_negotiation.cc:407] Connection header received
0924 16:11:56.371934 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:56.371941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:56.372067 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:56.372228 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:56.373844 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.375044 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.377004 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.377678 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.377829 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:56.382244 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:56.382252 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:56.382256 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:56.382311 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:56.386355 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:56.387280 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:56.387285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:56.387288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:56.387433 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:56.387810 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:56.387813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:56.387816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:56.388167 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:56.388350 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:56.388862 (+ 512us) server_negotiation.cc:299] Negotiation successful
0924 16:11:56.389127 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":285,"threads_started":1}
I20260924 16:11:56.389577 27223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:56.370222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.254:42565 (local address 127.0.0.1:49692)
0924 16:11:56.371474 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:56.371542 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:56.371769 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:56.372334 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:56.372341 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:56.373237 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:56.373648 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:56.373661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.375231 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.375237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:56.376825 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:56.376842 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.377988 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.377997 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:56.378253 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:56.379151 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:56.379170 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:56.381864 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:56.386553 (+ 4689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:56.386569 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:56.386576 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:56.387132 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:56.387556 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:56.387559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:56.387562 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:56.387702 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:56.388316 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:56.388326 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:56.388556 (+ 230us) client_negotiation.cc:769] Sending connection context
0924 16:11:56.388736 (+ 180us) client_negotiation.cc:240] Negotiation successful
0924 16:11:56.389142 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"thread_start_us":691,"threads_started":1}
I20260924 16:11:56.392144 27218 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:11:56 dist-test-slave-1gc3 krb5kdc[26989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.254@KRBTEST.COM
Sep 24 16:11:56 dist-test-slave-1gc3 krb5kdc[26989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266316, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.252@KRBTEST.COM for kudu/127.3.214.253@KRBTEST.COM
I20260924 16:11:56.404903 27224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:56.385392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49704 (local address 127.3.214.254:42565)
0924 16:11:56.386191 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:56.386196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:56.386223 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:11:56.386290 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:56.386293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:56.386369 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:56.386497 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:56.388012 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.388962 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.390543 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.391228 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.391374 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:56.396812 (+ 5438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:56.396820 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:56.396825 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:56.396885 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:56.400838 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:56.402082 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:56.402088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:56.402091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:56.402387 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:56.402954 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:56.402960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:56.402964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:56.403418 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:56.403732 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:56.404138 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 16:11:56.404459 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":349,"threads_started":1}
I20260924 16:11:56.410203 27225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:56.393317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.3.214.253:39981)
0924 16:11:56.393874 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:56.393879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:56.393905 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:11:56.393962 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:56.393966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:56.394031 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:56.394167 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:56.395099 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.395967 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.397437 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.398160 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.398356 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:56.402448 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:56.402458 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:56.402464 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:56.402523 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:56.406320 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:56.407265 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:56.407271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:56.407275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:56.407541 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:56.408366 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:56.408372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:56.408374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:56.408849 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:56.409013 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:56.409531 (+ 518us) server_negotiation.cc:299] Negotiation successful
0924 16:11:56.409802 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":296,"threads_started":1}
I20260924 16:11:56.410244 27223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:56.393151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.253:39981 (local address 127.0.0.1:56716)
0924 16:11:56.393332 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:56.393352 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:56.393467 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:56.394204 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:56.394208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:56.394668 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:56.394860 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:56.394873 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.396150 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.396157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:56.397232 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:56.397248 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.398446 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.398454 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:56.398644 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:56.399319 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:56.399339 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:56.402078 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:56.406511 (+ 4433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:56.406522 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:56.406526 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:56.407052 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:56.407719 (+ 667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:56.407724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:56.407727 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:56.408198 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:56.409017 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:56.409022 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:56.409250 (+ 228us) client_negotiation.cc:769] Sending connection context
0924 16:11:56.409429 (+ 179us) client_negotiation.cc:240] Negotiation successful
0924 16:11:56.409798 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 16:11:56.411439 27218 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:11:56 dist-test-slave-1gc3 krb5kdc[26989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.214.253@KRBTEST.COM
Sep 24 16:11:56 dist-test-slave-1gc3 krb5kdc[26989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266316, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.252@KRBTEST.COM for kudu/127.3.214.252@KRBTEST.COM
I20260924 16:11:56.426533 27226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:56.407590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56732 (local address 127.3.214.253:39981)
0924 16:11:56.408575 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:56.408581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:56.408607 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:11:56.408654 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:56.408658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:56.408735 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:56.408879 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:56.410772 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.411985 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.413739 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.414348 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.414484 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:56.420132 (+ 5648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:56.420139 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:56.420143 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:56.420192 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:56.423405 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:56.424215 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:56.424220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:56.424222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:56.424363 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:56.424743 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:56.424747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:56.424750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:56.425091 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:56.425382 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:56.425856 (+ 474us) server_negotiation.cc:299] Negotiation successful
0924 16:11:56.426067 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":830,"thread_start_us":572,"threads_started":1}
I20260924 16:11:56.428408 27221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:56.412949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38604 (local address 127.3.214.252:44423)
0924 16:11:56.413260 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:56.413266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:56.413312 (+ 46us) server_negotiation.cc:407] Connection header received
0924 16:11:56.413406 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:56.413412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:56.413534 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:56.413718 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:56.415181 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.415976 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.417188 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.417619 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.417730 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:56.421555 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:56.421562 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:56.421566 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:56.421619 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:56.424898 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:56.425823 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:56.425830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:56.425834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:56.426070 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:56.426615 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:56.426620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:56.426622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:56.427022 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:56.427178 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:56.427738 (+ 560us) server_negotiation.cc:299] Negotiation successful
0924 16:11:56.428015 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260924 16:11:56.428593 27223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:56.412682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:44423 (local address 127.0.0.1:38604)
0924 16:11:56.413115 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:56.413136 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:56.413240 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:56.414246 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:56.414252 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:56.414846 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:56.415022 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:56.415032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.416111 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.416118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:56.417044 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:56.417056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.417784 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.417790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:56.418009 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:56.418695 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:56.418721 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:56.421233 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:56.425126 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:56.425133 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:56.425137 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:56.425652 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:56.426276 (+ 624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:56.426281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:56.426285 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:56.426454 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:56.427236 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:56.427242 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:56.427460 (+ 218us) client_negotiation.cc:769] Sending connection context
0924 16:11:56.427662 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 16:11:56.428240 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":178}
I20260924 16:11:56.430846 27143 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } attempt: 5
I20260924 16:11:56.433425 3931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:11:56.436483 27218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481f86d1f74c44b2b5cafcf7347e5741: Bootstrap starting.
I20260924 16:11:56.436116 27044 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "64bf3d337d5548bd8bb165e102c93a63"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49704
Sep 24 16:11:56 dist-test-slave-1gc3 krb5kdc[26989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.253@KRBTEST.COM for kudu/127.3.214.252@KRBTEST.COM
I20260924 16:11:56.442443 27218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 481f86d1f74c44b2b5cafcf7347e5741: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:56.444133 27218 log.cc:824] T 00000000000000000000000000000000 P 481f86d1f74c44b2b5cafcf7347e5741: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:56.448948 27218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481f86d1f74c44b2b5cafcf7347e5741: No bootstrap required, opened a new log
I20260924 16:11:56.449681 27146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:56.431618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:44423 (local address 127.0.0.1:38606)
0924 16:11:56.431969 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:56.431994 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:56.432146 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:56.432676 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:56.432680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:56.433057 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:56.433230 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:56.433239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.434901 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.434909 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:56.436423 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:56.436451 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.437711 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.437718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:56.438015 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:56.438877 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:56.438907 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:56.442204 (+ 3297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:56.446855 (+ 4651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:56.446862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:56.446867 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:56.447344 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:56.447748 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:56.447753 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:56.447757 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:56.447905 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:56.448542 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:56.448547 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:56.448711 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 16:11:56.448847 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 16:11:56.449306 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260924 16:11:56.449667 27221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:56.431642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38606 (local address 127.3.214.252:44423)
0924 16:11:56.431975 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:56.431981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:56.432282 (+ 301us) server_negotiation.cc:407] Connection header received
0924 16:11:56.432353 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:56.432364 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:56.432453 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:56.432635 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:56.433403 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.434688 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.436660 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.437484 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.437646 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:56.442474 (+ 4828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:56.442480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:56.442483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:56.442525 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:56.446658 (+ 4133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:56.447475 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:56.447481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:56.447485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:56.447635 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:56.448014 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:56.448019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:56.448022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:56.448419 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:56.448580 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:56.449005 (+ 425us) server_negotiation.cc:299] Negotiation successful
0924 16:11:56.449239 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260924 16:11:56.455021 27143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3335f30abb40739b6e58da9db3563f: Bootstrap starting.
I20260924 16:11:56.459753 27143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f3335f30abb40739b6e58da9db3563f: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:56.460111 27218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 481f86d1f74c44b2b5cafcf7347e5741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64bf3d337d5548bd8bb165e102c93a63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42565 } } peers { permanent_uuid: "9f3335f30abb40739b6e58da9db3563f" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } } peers { permanent_uuid: "481f86d1f74c44b2b5cafcf7347e5741" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } }
I20260924 16:11:56.460677 27218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 481f86d1f74c44b2b5cafcf7347e5741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 481f86d1f74c44b2b5cafcf7347e5741, State: Initialized, Role: FOLLOWER
I20260924 16:11:56.461282 27143 log.cc:824] T 00000000000000000000000000000000 P 9f3335f30abb40739b6e58da9db3563f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:56.461373 27218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 481f86d1f74c44b2b5cafcf7347e5741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64bf3d337d5548bd8bb165e102c93a63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42565 } } peers { permanent_uuid: "9f3335f30abb40739b6e58da9db3563f" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } } peers { permanent_uuid: "481f86d1f74c44b2b5cafcf7347e5741" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } }
I20260924 16:11:56.462823 27229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481f86d1f74c44b2b5cafcf7347e5741 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64bf3d337d5548bd8bb165e102c93a63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42565 } } peers { permanent_uuid: "9f3335f30abb40739b6e58da9db3563f" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } } peers { permanent_uuid: "481f86d1f74c44b2b5cafcf7347e5741" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } } }
I20260924 16:11:56.463315 27229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481f86d1f74c44b2b5cafcf7347e5741 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:11:56.464471 27218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 481f86d1f74c44b2b5cafcf7347e5741 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:56.465179 27143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3335f30abb40739b6e58da9db3563f: No bootstrap required, opened a new log
I20260924 16:11:56.476207 27143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f3335f30abb40739b6e58da9db3563f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64bf3d337d5548bd8bb165e102c93a63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42565 } } peers { permanent_uuid: "9f3335f30abb40739b6e58da9db3563f" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } } peers { permanent_uuid: "481f86d1f74c44b2b5cafcf7347e5741" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } }
I20260924 16:11:56.476701 27143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f3335f30abb40739b6e58da9db3563f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f3335f30abb40739b6e58da9db3563f, State: Initialized, Role: FOLLOWER
I20260924 16:11:56.477264 27143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f3335f30abb40739b6e58da9db3563f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64bf3d337d5548bd8bb165e102c93a63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42565 } } peers { permanent_uuid: "9f3335f30abb40739b6e58da9db3563f" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } } peers { permanent_uuid: "481f86d1f74c44b2b5cafcf7347e5741" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } }
W20260924 16:11:56.477777 27240 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 481f86d1f74c44b2b5cafcf7347e5741: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:11:56.477998 27240 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:11:56.478991 27242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3335f30abb40739b6e58da9db3563f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64bf3d337d5548bd8bb165e102c93a63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42565 } } peers { permanent_uuid: "9f3335f30abb40739b6e58da9db3563f" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } } peers { permanent_uuid: "481f86d1f74c44b2b5cafcf7347e5741" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } } }
I20260924 16:11:56.479624 27242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3335f30abb40739b6e58da9db3563f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:11:56.480619 27143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f3335f30abb40739b6e58da9db3563f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:56.487504 27069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } attempt: 5
I20260924 16:11:56.474648 27151 master_runner.cc:186] Error getting master registration for 127.3.214.254:42565: OK, instance_id { permanent_uuid: "64bf3d337d5548bd8bb165e102c93a63" instance_seqno: 1790266315040607 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:11:56.488714 27151 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 24 16:11:56 dist-test-slave-1gc3 krb5kdc[26989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266314, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.214.254@KRBTEST.COM for kudu/127.3.214.252@KRBTEST.COM
I20260924 16:11:56.511724 27221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:56.489574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38612 (local address 127.3.214.252:44423)
0924 16:11:56.489843 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:56.489851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:56.489882 (+ 31us) server_negotiation.cc:407] Connection header received
0924 16:11:56.490073 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:56.490078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:56.490213 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:56.490340 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:56.491581 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.493047 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.495563 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.496379 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.496574 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:56.502005 (+ 5431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:56.502016 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:56.502022 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:56.502148 (+ 126us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:56.506825 (+ 4677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:56.508410 (+ 1585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:56.508417 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:56.508422 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:56.508654 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:56.509422 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:56.509429 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:56.509432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:56.509926 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:56.510073 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:56.510994 (+ 921us) server_negotiation.cc:299] Negotiation successful
0924 16:11:56.511293 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 16:11:56.513052 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:56.488520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.214.252:44423 (local address 127.0.0.1:38612)
0924 16:11:56.489049 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:56.489074 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:56.489221 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:56.490457 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:56.490463 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:56.491057 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:56.491344 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:56.491358 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.493398 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.493409 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:56.495275 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:56.495307 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.496621 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.496628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:56.496874 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:56.497784 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:56.497815 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:56.501608 (+ 3793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:56.507142 (+ 5534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:56.507150 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:56.507156 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:56.508198 (+ 1042us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:56.508808 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:56.508834 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:56.508838 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:56.509181 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:56.510307 (+ 1126us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:56.510312 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:56.510581 (+ 269us) client_negotiation.cc:769] Sending connection context
0924 16:11:56.511977 (+ 1396us) client_negotiation.cc:240] Negotiation successful
0924 16:11:56.512473 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354}
I20260924 16:11:56.509774 27076 master_runner.cc:186] Error getting master registration for 127.3.214.254:42565: OK, instance_id { permanent_uuid: "64bf3d337d5548bd8bb165e102c93a63" instance_seqno: 1790266315040607 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:11:56.520347 27076 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:11:56.521011 27069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63: Bootstrap starting.
I20260924 16:11:56.526599 27069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:56.528424 27069 log.cc:824] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63: Log is configured to *not* fsync() on all Append() calls
W20260924 16:11:56.530382 27254 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f3335f30abb40739b6e58da9db3563f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:11:56.530609 27254 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:11:56.533732 27069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63: No bootstrap required, opened a new log
I20260924 16:11:56.541990 27069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64bf3d337d5548bd8bb165e102c93a63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42565 } } peers { permanent_uuid: "9f3335f30abb40739b6e58da9db3563f" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } } peers { permanent_uuid: "481f86d1f74c44b2b5cafcf7347e5741" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } }
I20260924 16:11:56.542512 27069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64bf3d337d5548bd8bb165e102c93a63, State: Initialized, Role: FOLLOWER
I20260924 16:11:56.543082 27069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64bf3d337d5548bd8bb165e102c93a63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42565 } } peers { permanent_uuid: "9f3335f30abb40739b6e58da9db3563f" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } } peers { permanent_uuid: "481f86d1f74c44b2b5cafcf7347e5741" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } }
I20260924 16:11:56.544428 27257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64bf3d337d5548bd8bb165e102c93a63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42565 } } peers { permanent_uuid: "9f3335f30abb40739b6e58da9db3563f" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } } peers { permanent_uuid: "481f86d1f74c44b2b5cafcf7347e5741" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } } }
I20260924 16:11:56.545171 27257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:11:56.546092 27069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:56.546240 27044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 16:11:56.546427 27044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:56.550537 27044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64bf3d337d5548bd8bb165e102c93a63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42565 } } peers { permanent_uuid: "9f3335f30abb40739b6e58da9db3563f" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } } peers { permanent_uuid: "481f86d1f74c44b2b5cafcf7347e5741" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } }
I20260924 16:11:56.554152 27118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "64bf3d337d5548bd8bb165e102c93a63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9f3335f30abb40739b6e58da9db3563f"
I20260924 16:11:56.554911 27118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f3335f30abb40739b6e58da9db3563f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:56.560583 27118 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9f3335f30abb40739b6e58da9db3563f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64bf3d337d5548bd8bb165e102c93a63 in term 1.
I20260924 16:11:56.562726 27193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "64bf3d337d5548bd8bb165e102c93a63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "481f86d1f74c44b2b5cafcf7347e5741"
I20260924 16:11:56.563473 27193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 481f86d1f74c44b2b5cafcf7347e5741 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:56.568838 27193 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 481f86d1f74c44b2b5cafcf7347e5741 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64bf3d337d5548bd8bb165e102c93a63 in term 1.
I20260924 16:11:56.570127 27020 leader_election.cc:302] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 481f86d1f74c44b2b5cafcf7347e5741, 64bf3d337d5548bd8bb165e102c93a63; no voters:
I20260924 16:11:56.570914 27044 leader_election.cc:288] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63 [CANDIDATE]: Term 1 election: Requested vote from peers 9f3335f30abb40739b6e58da9db3563f (127.3.214.253:39981), 481f86d1f74c44b2b5cafcf7347e5741 (127.3.214.252:44423)
W20260924 16:11:56.575896 27172 tablet.cc:2369] T 00000000000000000000000000000000 P 481f86d1f74c44b2b5cafcf7347e5741: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:11:56.581198 27257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:56.590037 27257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63 [term 1 LEADER]: Becoming Leader. State: Replica: 64bf3d337d5548bd8bb165e102c93a63, State: Running, Role: LEADER
I20260924 16:11:56.592267 27257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64bf3d337d5548bd8bb165e102c93a63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42565 } } peers { permanent_uuid: "9f3335f30abb40739b6e58da9db3563f" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } } peers { permanent_uuid: "481f86d1f74c44b2b5cafcf7347e5741" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } }
I20260924 16:11:56.623651 27261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64bf3d337d5548bd8bb165e102c93a63. Latest consensus state: current_term: 1 leader_uuid: "64bf3d337d5548bd8bb165e102c93a63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64bf3d337d5548bd8bb165e102c93a63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42565 } } peers { permanent_uuid: "9f3335f30abb40739b6e58da9db3563f" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } } peers { permanent_uuid: "481f86d1f74c44b2b5cafcf7347e5741" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } } }
I20260924 16:11:56.624395 27261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:56.634460 27269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:56.658627 27221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:56.608672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38622 (local address 127.3.214.252:44423)
0924 16:11:56.608960 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:56.608968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:56.619074 (+ 10106us) server_negotiation.cc:407] Connection header received
0924 16:11:56.619469 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:56.619476 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:56.619592 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:56.619680 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:56.631735 (+ 12055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.633020 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.635175 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.635967 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.636089 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:56.640619 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:56.640627 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:56.640633 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:56.640703 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:56.644805 (+ 4102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:56.654393 (+ 9588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:56.654403 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:56.654408 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:56.654713 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:56.655451 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:56.655457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:56.655461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:56.655955 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:56.656313 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:56.657673 (+ 1360us) server_negotiation.cc:299] Negotiation successful
0924 16:11:56.658071 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
W20260924 16:11:56.673053 27097 tablet.cc:2369] T 00000000000000000000000000000000 P 9f3335f30abb40739b6e58da9db3563f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:11:56.677024 27269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:56.686275 27224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:56.602561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49710 (local address 127.3.214.254:42565)
0924 16:11:56.603713 (+ 1152us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:56.603721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:56.618817 (+ 15096us) server_negotiation.cc:407] Connection header received
0924 16:11:56.619274 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:56.619282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:56.619424 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:56.619533 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:56.638538 (+ 19005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.639820 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.646208 (+ 6388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.649734 (+ 3526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.649883 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:56.657823 (+ 7940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:56.657836 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:56.657843 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:56.657901 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:56.662742 (+ 4841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:56.677407 (+ 14665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:56.677427 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:56.677432 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:56.677703 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:56.681340 (+ 3637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:56.681349 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:56.681353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:56.681875 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:56.682259 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:56.685465 (+ 3206us) server_negotiation.cc:299] Negotiation successful
0924 16:11:56.685829 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320}
I20260924 16:11:56.691517 27226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:56.614520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56742 (local address 127.3.214.253:39981)
0924 16:11:56.614819 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:56.614827 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:56.629761 (+ 14934us) server_negotiation.cc:407] Connection header received
0924 16:11:56.630514 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:56.630522 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:56.630637 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:56.630719 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:56.642681 (+ 11962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.643863 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.646804 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:56.647671 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:56.647786 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:56.664191 (+ 16405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:56.664200 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:56.664206 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:56.664271 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:56.668987 (+ 4716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:56.682446 (+ 13459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:56.682454 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:56.682458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:56.682704 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:56.683803 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:56.683810 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:56.683813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:56.684224 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:56.684786 (+ 562us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:56.690637 (+ 5851us) server_negotiation.cc:299] Negotiation successful
0924 16:11:56.691042 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260924 16:11:56.700169 27279 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 16:11:56.740185 27193 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 481f86d1f74c44b2b5cafcf7347e5741 [term 1 FOLLOWER]: Refusing update from remote peer 64bf3d337d5548bd8bb165e102c93a63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:56.744586 27118 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9f3335f30abb40739b6e58da9db3563f [term 1 FOLLOWER]: Refusing update from remote peer 64bf3d337d5548bd8bb165e102c93a63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:56.750346 27257 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f3335f30abb40739b6e58da9db3563f" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:11:56.751492 27257 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "481f86d1f74c44b2b5cafcf7347e5741" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 16:11:56.778054 27023 tablet.cc:2369] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:11:56.805883 27229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481f86d1f74c44b2b5cafcf7347e5741 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64bf3d337d5548bd8bb165e102c93a63. Latest consensus state: current_term: 1 leader_uuid: "64bf3d337d5548bd8bb165e102c93a63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64bf3d337d5548bd8bb165e102c93a63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42565 } } peers { permanent_uuid: "9f3335f30abb40739b6e58da9db3563f" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } } peers { permanent_uuid: "481f86d1f74c44b2b5cafcf7347e5741" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } } }
I20260924 16:11:56.806414 27229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481f86d1f74c44b2b5cafcf7347e5741 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:11:56.812233 27261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64bf3d337d5548bd8bb165e102c93a63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64bf3d337d5548bd8bb165e102c93a63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42565 } } peers { permanent_uuid: "9f3335f30abb40739b6e58da9db3563f" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } } peers { permanent_uuid: "481f86d1f74c44b2b5cafcf7347e5741" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } } }
I20260924 16:11:56.812752 27261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:56.814483 27242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3335f30abb40739b6e58da9db3563f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64bf3d337d5548bd8bb165e102c93a63. Latest consensus state: current_term: 1 leader_uuid: "64bf3d337d5548bd8bb165e102c93a63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64bf3d337d5548bd8bb165e102c93a63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42565 } } peers { permanent_uuid: "9f3335f30abb40739b6e58da9db3563f" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } } peers { permanent_uuid: "481f86d1f74c44b2b5cafcf7347e5741" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } } }
I20260924 16:11:56.814937 27242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3335f30abb40739b6e58da9db3563f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:11:56.830826 27229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481f86d1f74c44b2b5cafcf7347e5741 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64bf3d337d5548bd8bb165e102c93a63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64bf3d337d5548bd8bb165e102c93a63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42565 } } peers { permanent_uuid: "9f3335f30abb40739b6e58da9db3563f" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } } peers { permanent_uuid: "481f86d1f74c44b2b5cafcf7347e5741" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } } }
I20260924 16:11:56.831208 27229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481f86d1f74c44b2b5cafcf7347e5741 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:11:56.832103 27257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64bf3d337d5548bd8bb165e102c93a63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64bf3d337d5548bd8bb165e102c93a63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42565 } } peers { permanent_uuid: "9f3335f30abb40739b6e58da9db3563f" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } } peers { permanent_uuid: "481f86d1f74c44b2b5cafcf7347e5741" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } } }
I20260924 16:11:56.832530 27257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:56.835616 27242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3335f30abb40739b6e58da9db3563f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64bf3d337d5548bd8bb165e102c93a63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64bf3d337d5548bd8bb165e102c93a63" member_type: VOTER last_known_addr { host: "127.3.214.254" port: 42565 } } peers { permanent_uuid: "9f3335f30abb40739b6e58da9db3563f" member_type: VOTER last_known_addr { host: "127.3.214.253" port: 39981 } } peers { permanent_uuid: "481f86d1f74c44b2b5cafcf7347e5741" member_type: VOTER last_known_addr { host: "127.3.214.252" port: 44423 } } }
I20260924 16:11:56.836004 27242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3335f30abb40739b6e58da9db3563f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:11:56.843120 27269 catalog_manager.cc:1347] Generated new cluster ID: e73a33dbf3664755a6ad8b49cab90740
I20260924 16:11:56.843266 27269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:56.870476 27269 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:56.873339 27269 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:56.907416 27269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64bf3d337d5548bd8bb165e102c93a63: Generated new TSK 0
I20260924 16:11:56.908964 27269 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:11:57.081126 27224 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:57.040103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49718 (local address 127.3.214.254:42565)
0924 16:11:57.040382 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:57.040389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:57.073769 (+ 33380us) server_negotiation.cc:407] Connection header received
0924 16:11:57.078694 (+ 4925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:57.078703 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:57.078850 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:57.078960 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:57.080759 (+ 1799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49718: BlockingRecv error: recv got EOF from 127.0.0.1:49718 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 16:11:57.092756 27221 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:57.059029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38632 (local address 127.3.214.252:44423)
0924 16:11:57.059378 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:57.059386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:57.084416 (+ 25030us) server_negotiation.cc:407] Connection header received
0924 16:11:57.090385 (+ 5969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:57.090393 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:57.090508 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:57.090601 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:57.092396 (+ 1795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38632: BlockingRecv error: recv got EOF from 127.0.0.1:38632 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
W20260924 16:11:57.094878 27226 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:57.071962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56756 (local address 127.3.214.253:39981)
0924 16:11:57.072226 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:57.072274 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:57.085739 (+ 13465us) server_negotiation.cc:407] Connection header received
0924 16:11:57.086287 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:57.086295 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:57.086449 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:57.086552 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:57.094531 (+ 7979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56756: BlockingRecv error: recv got EOF from 127.0.0.1:56756 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260924 16:11:57.119980 27226 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:57.116476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56768 (local address 127.3.214.253:39981)
0924 16:11:57.116735 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:57.116744 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:57.117486 (+ 742us) server_negotiation.cc:407] Connection header received
0924 16:11:57.117970 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:57.117977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:57.118220 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:57.118313 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:57.119683 (+ 1370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56768: BlockingRecv error: recv got EOF from 127.0.0.1:56768 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260924 16:11:57.132400 27226 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:57.128986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56774 (local address 127.3.214.253:39981)
0924 16:11:57.129271 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:57.129279 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:57.130076 (+ 797us) server_negotiation.cc:407] Connection header received
0924 16:11:57.130518 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:57.130527 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:57.130677 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:57.130771 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:57.132020 (+ 1249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56774: BlockingRecv error: recv got EOF from 127.0.0.1:56774 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260924 16:11:57.145507 27226 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:57.140660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56790 (local address 127.3.214.253:39981)
0924 16:11:57.140939 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:57.140947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:57.143059 (+ 2112us) server_negotiation.cc:407] Connection header received
0924 16:11:57.143472 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:57.143480 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:57.143606 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:57.143710 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:57.145148 (+ 1438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56790: BlockingRecv error: recv got EOF from 127.0.0.1:56790 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
W20260924 16:11:57.158864 27226 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:57.155524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56792 (local address 127.3.214.253:39981)
0924 16:11:57.155803 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:57.155811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:57.156529 (+ 718us) server_negotiation.cc:407] Connection header received
0924 16:11:57.157012 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:57.157019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:57.157158 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:57.157263 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:57.158508 (+ 1245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56792: BlockingRecv error: recv got EOF from 127.0.0.1:56792 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
W20260924 16:11:57.176725 27226 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:57.173140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56802 (local address 127.3.214.253:39981)
0924 16:11:57.173401 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:57.173408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:57.174265 (+ 857us) server_negotiation.cc:407] Connection header received
0924 16:11:57.174749 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:57.174757 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:57.174922 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:57.175031 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:57.176348 (+ 1317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56802: BlockingRecv error: recv got EOF from 127.0.0.1:56802 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260924 16:11:57.202291 27226 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:57.198735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56812 (local address 127.3.214.253:39981)
0924 16:11:57.199111 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:57.199119 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:57.199828 (+ 709us) server_negotiation.cc:407] Connection header received
0924 16:11:57.200114 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:57.200121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:57.200241 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:57.200411 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:57.201938 (+ 1527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56812: BlockingRecv error: recv got EOF from 127.0.0.1:56812 (error 108)
Metrics: {"server-negotiator.queue_time_us":179}
W20260924 16:11:57.241961 27226 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:57.238794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56818 (local address 127.3.214.253:39981)
0924 16:11:57.239040 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:57.239045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:57.239637 (+ 592us) server_negotiation.cc:407] Connection header received
0924 16:11:57.240018 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:57.240053 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:57.240189 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:57.240431 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:57.241666 (+ 1235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56818: BlockingRecv error: recv got EOF from 127.0.0.1:56818 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260924 16:11:57.313145 27226 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:57.310922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56834 (local address 127.3.214.253:39981)
0924 16:11:57.311229 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:57.311236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:57.311496 (+ 260us) server_negotiation.cc:407] Connection header received
0924 16:11:57.311761 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:57.311765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:57.311867 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:57.311963 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:57.312895 (+ 932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56834: BlockingRecv error: recv got EOF from 127.0.0.1:56834 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
W20260924 16:11:57.448423 27226 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:57.445733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56836 (local address 127.3.214.253:39981)
0924 16:11:57.446020 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:57.446031 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:57.446556 (+ 525us) server_negotiation.cc:407] Connection header received
0924 16:11:57.446787 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:57.446792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:57.446891 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:57.447017 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:57.448112 (+ 1095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56836: BlockingRecv error: recv got EOF from 127.0.0.1:56836 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 16:11:57.480289 27240 catalog_manager.cc:1259] Loaded cluster ID: e73a33dbf3664755a6ad8b49cab90740
I20260924 16:11:57.480413 27240 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 481f86d1f74c44b2b5cafcf7347e5741: loading cluster ID for follower catalog manager: success
I20260924 16:11:57.483304 27240 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 481f86d1f74c44b2b5cafcf7347e5741: acquiring CA information for follower catalog manager: success
I20260924 16:11:57.485400 27240 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 481f86d1f74c44b2b5cafcf7347e5741: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:11:57.533164 27254 catalog_manager.cc:1259] Loaded cluster ID: e73a33dbf3664755a6ad8b49cab90740
I20260924 16:11:57.533298 27254 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9f3335f30abb40739b6e58da9db3563f: loading cluster ID for follower catalog manager: success
I20260924 16:11:57.536327 27254 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9f3335f30abb40739b6e58da9db3563f: acquiring CA information for follower catalog manager: success
I20260924 16:11:57.538749 27254 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9f3335f30abb40739b6e58da9db3563f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:11:57.716478 27226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:57.709250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56842 (local address 127.3.214.253:39981)
0924 16:11:57.709511 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:57.709517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:57.709951 (+ 434us) server_negotiation.cc:407] Connection header received
0924 16:11:57.710166 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:57.710173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:57.710424 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:57.710600 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:57.711171 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:57.712074 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:57.713589 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:57.714058 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:57.714207 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:57.714933 (+ 726us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:57.715678 (+ 745us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:57.715815 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:57.716069 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 16:11:57.716185 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 16:11:57.730875 27352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:57.724114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38638 (local address 127.3.214.252:44423)
0924 16:11:57.724730 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:57.724735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:57.724760 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:11:57.724979 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:57.724983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:57.725170 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:57.725349 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:57.725866 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:57.726736 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:57.728276 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:57.728719 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:57.728888 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:57.729394 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:57.730070 (+ 676us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:57.730258 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:57.730376 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 16:11:57.730465 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":313,"threads_started":1}
I20260924 16:11:57.759030 27226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:57.738576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56846 (local address 127.3.214.253:39981)
0924 16:11:57.738833 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:57.738841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:57.752140 (+ 13299us) server_negotiation.cc:407] Connection header received
0924 16:11:57.752341 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:57.752348 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:57.752477 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:57.752604 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:57.753500 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:57.754616 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:57.756678 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:57.757348 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:57.757493 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:57.758066 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:57.758301 (+ 235us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:57.758383 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:57.758575 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 16:11:57.758677 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 16:11:57.759084 27358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:57.738130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49732 (local address 127.3.214.254:42565)
0924 16:11:57.751326 (+ 13196us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:57.751334 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:57.751362 (+ 28us) server_negotiation.cc:407] Connection header received
0924 16:11:57.751530 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:57.751536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:57.751866 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:57.752098 (+ 232us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:57.753003 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:57.754070 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:57.756180 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:57.756833 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:57.756966 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:57.757501 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:57.758025 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:57.758141 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:57.758337 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 16:11:57.758470 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13039,"thread_start_us":13234,"threads_started":1}
I20260924 16:11:57.759202 27352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:57.738296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38642 (local address 127.3.214.252:44423)
0924 16:11:57.738502 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:57.738510 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:57.752926 (+ 14416us) server_negotiation.cc:407] Connection header received
0924 16:11:57.753060 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:57.753067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:57.753161 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:57.753266 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:57.753871 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:57.754864 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:57.756883 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:57.757613 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:57.757741 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:57.758349 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:57.758482 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:57.758574 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:57.758800 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 16:11:57.758897 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 16:11:57.766196 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 27002
I20260924 16:11:57.767715 27064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:57.886368 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 27002
I20260924 16:11:57.912001 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 27076
I20260924 16:11:57.913527 27138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:58.030179 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 27076
I20260924 16:11:58.053884 3931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu pid 27151
I20260924 16:11:58.055400 27213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:11:58.182515 3931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHlT5a5/build/asan/bin/kudu with pid 27151
2026-09-24T16:11:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5116 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10227 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (397527 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST